System and method for acquiring, editing, generating and outputting video data

ABSTRACT

Systems and methods directed to acquiring, generating, manipulating and/or editing refocusable video data/frames. The refocusable video frames may be light field video frames that may be focused and/or re-focused after acquisition or recording of such video frames. In one aspect, method comprises: selecting a first key frame, wherein the first key frame corresponds to one of a plurality of refocusable light field video frames, selecting a second key frame, wherein the second key frame corresponds to one of the plurality of refocusable light field video frames which is temporally spaced apart from the first key frame such that a plurality of refocusable light field video frames are temporally disposed between the first and the second key frames, determining a virtual focus parameters for the first key frame and the second key frame, and generating first video data.

RELATED APPLICATION

This application claims priority to U.S. Provisional Application Ser.No. 61/117,621, entitled “System of and Method for Video Refocusing”,filed Nov. 25, 2008; U.S. Provisional Application Ser. No. 61/120,530,entitled “Light Field Camera and System, and Methods of Using andManufacturing Same”, filed Dec. 8, 2008; and U.S. ProvisionalApplication Ser. No. 61/170,620, entitled “Light Field Camera Image,File and Configuration Data, and Method of Using, Storing andCommunicating Same”, filed Apr. 18, 2009. The contents of theseprovisional applications are incorporated by reference herein, in theirentirety.

INTRODUCTION

In one aspect, the present inventions are directed to, among otherthings, systems for and methods of acquiring, generating, manipulatingand/or editing (for example, focusing or refocusing) refocusable videodata, information, images and/or frames. Notably, refocusable videodata, information, images and/or frames are video data, information,images and/or frames, no matter how acquired or obtained, that may befocused and/or re-focused after acquisition or recording of the data,information, images and/or frames. For example, in one embodiment,refocusable video data, information, images and/or frames are lightfield data, information, images and/or frames acquired or obtained, forexample, via a light field acquisition system or device.

Briefly, in the context of cinematography, post-production in thisapplication may mean the work and/or effort expended to place one ormore frames in a predetermined, desired and/or final form (for example,for viewing) after the acquisition or recording phase. This includes,for example, editing and special effects, of, for example, data which isrepresentative of 2D video. Notably, 2D video is video data which iscapable of or designed for viewing on two dimensional viewing devices.2D video formats include, but are not limited to, a signal suitable fortelevision viewing, film, or video files used for playback on a generalpurpose computing system.

There are many inventions described and illustrated herein. Some of thepresent inventions are described, illustrated and/or organized in thecontext of four basic sections or groups—which may be generallydescribed as: (1) systems and techniques of focusing video based on keyframes in post-production, (2) systems and techniques of focusing videobased on tracking subjects, (3) an indicator to provide information (forexample, to an operator during a “live” shoot) regarding the refocusablerange, or the extent thereof, in the video scene; such indicator mayallow the operator to determine, assess and/or understand the range ofrefocusing in the video scene, or the extent of refocusability thereof,before, during or after capture or acquisition of video of a scene, and(4) systems and techniques for automatically focusing video on theacquisition device using, in part, frames acquired temporally after theframe which is being focused. Briefly, setting the focus based on keyframes in post-production includes, for example, selecting the focusdepth for one or more key or predetermined frames in the videostream—after refocusable video has been captured or acquired with arefocusable video capture device. The focus depth for intermediateframes may be based on or determined using the focus depth of one ormore of the key or predetermined frames. For example, the focus depthfor intermediate frames may be interpolated from the focus depth of thekey or predetermined frames.

The section pertaining to setting the focus based on tracking subjectsmay include selecting one or more focuses to track a particular orpredetermined subject in, for example, a scene and during refocusablevideo editing. A subject is selected in one subject key frame, and thevideo editing system will keep that subject in focus during the videountil a new focus or subject key frame changes the focus.

Further, the section relating to an indicator used during a live shootthat allows the operator to understand the sharply refocusable range inthe scene may be characterized as when acquiring or recordingrefocusable video, the camera or focus operator/user is shown a live ornear live display that indicates the range of subjects that can bebrought into sharp focus. The operator/user may be shown a computed viewof the scene with a low effective aperture size that results in a depthof field that accurately represents the refocusable range of the videostream. The user or operator may, in response to the indication of therange of subjects that can be brought into sharp focus, change the oneor more setting or parameters of the camera or acquisition thereby.

As noted above, one section relates to systems and techniques to focusvideo data or frames (for example, automatically focus data or frames)on the acquisition device (or thereafter) using video data or framesacquired temporally after the frame(s) which is/are being focused orrefocused. In one embodiment, the systems and techniques include storingdata which is representative of a plurality of refocusable video frames,and using data of these frames or information derived from the data ofthese frames, the systems and techniques select one or more depths offocus for one or more video frames that were acquired and/or occurredearlier in time. The device includes a system for storing somerefocusable video, systems and techniques for analyzing or reviewingrefocusable frames for content, and systems and techniques for storingvideo (for example, 2D video) using the refocusable frames and selectedfocus.

Importantly, the present inventions are neither limited to any singleaspect nor embodiment, nor to any combinations and/or permutations ofsuch aspects and/or embodiments. Moreover, each of the aspects of thepresent inventions, and/or embodiments thereof, may be employed alone orin combination with one or more of the other aspects of the presentinventions and/or embodiments thereof. For the sake of brevity, many ofthose permutations and combinations will not be discussed and/orillustrated separately herein.

Notably, although the inventions are often described in the context of alight field capture system or device, which acquire or obtainrefocusable video data, information, images and/or frames, and/orprocesses or methods of acquiring, generating, manipulating and/orediting such refocusable video data, information, images and/or frames,it should be clear that the inventions are applicable to other systems,devices, processes and/or methods of acquiring, generating, manipulatingand/or editing refocusable video data. The refocusable video dataacquisition system may be systems/devices aimed at cinema professionals,including complete focus control after the video is captured, and/orsystems/devices directed to non-professionals including, for example,such devices as digital SLR camera for high-end consumers or consumervideo acquisition systems that perform automatic or semi-automatic focusadjustment controls and circuitry during the acquisition of the video.

For example, the present inventions may be implemented in conjunctionwith light field data acquisition devices (for example, cameras) and/orsystems to generate, manipulate and/or edit Light Field Data (forexample, adjust, select, define and/or redefine the focus and/or depthof field—after initial acquisition, capture and/or recording of theLight Field Data and/or information) of, for example, a scene. (See, forexample, United States Patent Application Publication 2007/0252074, andthe provisional applications to which it claims priority (namely, Ser.Nos. 60/615,179 and 60/647,492), and Ren Ng's PhD dissertation, “DigitalLight Field Photography”, Stanford University 2006, all of which areincorporated herein their entirety by reference; see also, the blockdiagram illustrations of exemplary light field acquisition devices inFIGS. 1A, 1B and 1D). Indeed, such light field data acquisition devicesmay be implemented in conjunction with post-processing circuitry whichis disposed in/on the acquisition device (see FIGS. 1C and 1F) orexternal thereto (see FIGS. 1C and 1F).

Notably, the term “Light Field Data” means data representing thelighting in a scene including, for example, a set of values, where eachvalue represents the light traveling along each geometric light ray (orbundle of rays approximating a geometric light ray) within acorresponding set of light rays. In a specific exemplary embodiment,Light Field Data represents the 2D image data read from the image sensorpixel array in a light field acquisition device (for example, a lightfield camera comprising a main lens, microlens array and a photo sensoras illustrated in FIGS. 1A-1F; see also, light field cameras illustratedin United States Patent Application Publication 2007/0252074, and/or theprovisional application to which it claims priority, and/or Ren Ng's PhDdissertation, “Digital Light Field Photography”, Stanford University2006). In another exemplary embodiment, Light Field Data means afunction L(x,y,u,v) where L is the amount of light (e.g. radiance)traveling along a ray (x,y,u,v) that passes through the optical apertureof the camera lens at 2D position (u,v) and the sensor at 2D position(x,y). In another exemplary embodiment, Light Field Data may mean theimage data collected with a coded aperture system (See FIG. 1H). In yetanother exemplary embodiment, Light Field Data may mean data encodedand/or recorded in the frequency spectrum of the light field. In yetanother exemplary embodiment, Light Field Data may mean a collection ofimages focused at different depths. In yet another exemplary embodiment,Light Field Data may mean a collection of images from differentviewpoints (See FIG. 1G). In yet another exemplary embodiment, LightField Data may mean a collection of images focused at a different set ofdepths in a scene. Notably, the foregoing are intended only as examplesof Light Field Data, and Light Field Data may in certain embodimentsinclude any collection of images or lighting data that may be used togenerate, derive, calculate, estimate and/or determine a full or partialrepresentation or approximation of a light field function L(x,y,u,v) asdescribed above.

Further, the term “circuit” may mean, among other things, a singlecomponent (for example, electrical/electronic) or a multiplicity ofcomponents (whether in integrated circuit form, discrete form orotherwise), which are active and/or passive, and which are coupledtogether to provide or perform a desired function. The term “circuitry”may mean, among other things, a circuit (whether integrated, discrete orotherwise), a group of such circuits, one or more processors, one ormore state machines, one or more processors implementing software, or acombination of one or more circuits (whether integrated, discrete orotherwise), one or more state machines, one or more processors, and/orone or more processors implementing software. Moreover, the term“optics” means one or more components and/or a system comprising aplurality of components used to affect the propagation of light,including but not limited to lens elements, windows, microlens arrays,apertures and mirrors.

SUMMARY OF CERTAIN ASPECTS OF THE INVENTIONS

There are many inventions described and illustrated herein. The presentinventions are neither limited to any single aspect nor embodimentthereof, nor to any combinations and/or permutations of such aspectsand/or embodiments. Moreover, each of the aspects of the presentinventions, and/or embodiments thereof, may be employed alone or incombination with one or more of the other aspects of the presentinventions and/or embodiments thereof. For the sake of brevity, many ofthose permutations and combinations will not be discussed separatelyherein.

In a first principal aspect, certain of the present inventions aredirected to a method of generating and outputting video datacorresponding to a plurality of video data frames, the method comprises:(a) receiving refocusable light field video data, wherein therefocusable light field video data includes a plurality of temporallycontiguous refocusable light field video frames, wherein eachrefocusable light field video frame includes an optical focus, (b)selecting a first key frame, wherein the first key frame corresponds toone of the plurality of refocusable light field video frames, (c)determining a virtual focus parameter for the first key frame, (d)selecting a second key frame, wherein the second key frame correspondsto one of the plurality of refocusable light field video frames which istemporally spaced apart from the first key frame such that a pluralityof refocusable light field video frames are temporally disposed betweenthe first and the second key frames, and (e) determining a virtual focusparameter for the second key frame.

The method of this aspect further includes generating first video datacorresponding to the plurality of refocusable light field video frameswhich are temporally disposed between the first and the second keyframes using (i) the virtual focus parameter for the first key frame,(ii) the virtual focus parameter for the second key frame and (iii) therefocusable light field video data corresponding to the plurality ofrefocusable light field video frames which are temporally disposedbetween the first and the second key frames, wherein the first videodata includes a plurality of video frames, each video frame including avirtual focus depth which is based on the virtual focus parameter forthe first key frame and/or virtual focus parameter for the second keyframe. The method may also include outputting the first video data.

The virtual focus parameter of the first key frame and the virtual focusparameter of the second key frame may each include information which isrepresentative of a virtual focus depth. Alternatively, the virtualfocus parameter of the first key frame and the virtual focus parameterof the second key frame each include information which is representativeof a location of a virtual focal plane, aperture and/or focus effect.Notably, the virtual focus depth corresponding to the virtual focusparameter of the first key frame may be different from the optical focusdepth of the refocusable light field video frame corresponding to thefirst key frame.

In one embodiment, the virtual focus parameter of the first key frameincludes information which is representative of focus tracking. Inanother embodiment, the virtual focus parameter of the first key frameincludes information which provides a tilted virtual focal plane.Notably, the method may also include generating final output video databy processing the first video data to provide compositing and/ortransitions and/or filtering and/or color adjustments of a plurality ofthe video frames corresponding to the first video data.

In another principal aspect, the present inventions are directed to amethod of generating and outputting video data corresponding to aplurality of video data frames, the method comprises: (a) receivingrefocusable light field video data, wherein the refocusable light fieldvideo data includes a plurality of temporally contiguous refocusablelight field video frames, wherein each refocusable light field videoframe includes an optical focus, (b) selecting a first key frame,wherein the first key frame corresponds to one of the plurality ofrefocusable light field video frames, (c) determining a virtual focusdepth for the first key frame, wherein the virtual focus depth for thefirst key frame is different from the optical focus depth of therefocusable light field video frame corresponding to the first keyframe, (d) selecting a second key frame, wherein the second key framecorresponds to one of the plurality of refocusable light field videoframes which is temporally spaced apart from the first key frame suchthat a plurality of refocusable light field video frames are temporallydisposed between the first and the second key frames, (e) determining avirtual focus depth for the second key frame, wherein the virtual focusdepth for the second key frame is different from the optical focus depthof the refocusable light field video frame corresponding to the secondkey frame, and (f) generating first video data corresponding to theplurality of refocusable light field video frames which are temporallydisposed between the first and the second key frames using (i) thevirtual focus depth for the first key frame, (ii) the virtual focusdepth for the second key frame, (iii) an inter-frame interpolation, and(iv) the refocusable light field video data corresponding to theplurality of refocusable light field video frames which are temporallydisposed between the first and the second key frames, wherein the firstvideo data includes a plurality of video frames, each video frameincluding a virtual focus depth which is based on the virtual focusdepth for the first key frame, the virtual focus depth for the secondkey frame, and inter-frame interpolation. The method of this aspect ofthe present inventions may also include outputting the first video data.

In one embodiment, the inter-frame interpolation includes linearinterpolation. In another embodiment the inter-frame interpolationincludes non-linear interpolation.

The virtual focus depth of the first key frame may provide a tiltedvirtual focal plane. The virtual focus depth of the first key frame andthe virtual focus depth of the second key frame each may provide atilted virtual focal plane. Indeed, a virtual focus depth of at leastone video frame of the first video data may provide a tilted focalplane.

In one embodiment, the method further includes inputting the inter-frameinterpolation. In another embodiment, the method further includesgenerating final output video data by processing the first video data toprovide compositing and/or transitions and/or filtering and/or coloradjustments of a plurality of the video frames corresponding to thefirst video data.

In another principal aspect, the present inventions are directed to asystem for editing video refocusable light field video data, includingfirst memory to store refocusable light field video data, wherein therefocusable light field video data includes a plurality of temporallycontiguous refocusable light field video frames, wherein eachrefocusable light field video frame includes an optical focus, and auser interface to receive input data which is representative of a firstkey frame, a second key frame, and inter-frame interpolation. The systemof this aspect of the present inventions also includes processingcircuitry, coupled to the first memory and the user interface to: (a)determine a virtual focus depth for the first key frame, wherein thevirtual focus depth for the first key frame is different from theoptical focus depth of the refocusable light field video framecorresponding to the first key frame, (b) determine a virtual focusdepth for the second key frame, wherein the virtual focus depth for thesecond key frame is different from the optical focus depth of therefocusable light field video frame corresponding to the second keyframe, and (c) generate first video data corresponding to the pluralityof refocusable light field video frames which are temporally disposedbetween the first and the second key frames using (i) the virtual focusdepth for the first key frame, (ii) the virtual focus depth for thesecond key frame, (iii) an inter-frame interpolation, and (iv) therefocusable light field video data corresponding to the plurality ofrefocusable light field video frames which are temporally disposedbetween the first and the second key frames, wherein the first videodata includes a plurality of video frames, each video frame including avirtual focus depth which is based on the virtual focus depth for thefirst key frame, the virtual focus depth for the second key frame, andinter-frame interpolation.

In one embodiment, the system further includes second memory, coupled tothe processing circuitry, wherein the second memory stores the firstvideo data. Moreover, the user interface may include a display tovisually output a plurality of the refocusable light field video frames.

Notably, the processing circuitry may also generate final output videoby processing the first video data to provide compositing and/ortransitions and/or filtering and/or color adjustments of a plurality ofthe video frames corresponding to the first video data.

In one embodiment, the inter-frame interpolation includes linearinterpolation. In another embodiment, the inter-frame interpolationincludes non-linear interpolation.

The virtual focus depth of the first key frame may provide a tiltedvirtual focal plane. Moreover, in another embodiment, a virtual focusdepth of at least one video frame of the first video data provides atilted virtual focal plane.

In another principal aspect, the present inventions are directed to asystem for editing video refocusable light field video data, systemcomprising:

means for storing refocusable light field video data, wherein therefocusable light field video data includes a plurality of temporallycontiguous refocusable light field video frames, wherein eachrefocusable light field video frame includes an optical focus;

means for receiving input data which is representative of a first keyframe, a second key frame, and inter-frame interpolation;

means for:

-   -   determining a virtual focus depth for the first key frame,        wherein the virtual focus depth for the first key frame is        different from the optical focus depth of the refocusable light        field video frame corresponding to the first key frame;    -   determining a virtual focus depth for the second key frame,        wherein the virtual focus depth for the second key frame is        different from the optical focus depth of the refocusable light        field video frame corresponding to the second key frame; and    -   generating first video data corresponding to the plurality of        refocusable light field video frames which are temporally        disposed between the first and the second key frames using (i)        the virtual focus depth for the first key frame, (ii) the        virtual focus depth for the second key frame, (iii) an        inter-frame interpolation, and (iv) the refocusable light field        video data corresponding to the plurality of refocusable light        field video frames which are temporally disposed between the        first and the second key frames, wherein the first video data        includes a plurality of video frames, each video frame including        a virtual focus depth which is based on the virtual focus depth        for the first key frame, the virtual focus depth for the second        key frame, and inter-frame interpolation.

Again, there are many inventions, and aspects of the inventions,described and illustrated herein. This Summary is not exhaustive of thescope of the present inventions. Indeed, this Summary may not bereflective of or correlate to the inventions protected by the claims inthis or in continuation/divisional applications hereof.

Moreover, this Summary is not intended to be limiting of the inventionsor the claims (whether the currently presented claims or claims of adivisional/continuation application) and should not be interpreted inthat manner. While certain embodiments have been described and/oroutlined in this Summary, it should be understood that the presentinventions are not limited to such embodiments, description and/oroutline, nor are the claims limited in such a manner (which should alsonot be interpreted as being limited by this Summary).

Indeed, many other aspects, inventions and embodiments, which may bedifferent from and/or similar to, the aspects, inventions andembodiments presented in this Summary, will be apparent from thedescription, illustrations and claims, which follow. In addition,although various features, attributes and advantages have been describedin this Summary and/or are apparent in light thereof, it should beunderstood that such features, attributes and advantages are notrequired whether in one, some or all of the embodiments of the presentinventions and, indeed, need not be present in any of the embodiments ofthe present inventions.

BRIEF DESCRIPTION OF THE DRAWINGS

In the course of the detailed description to follow, reference will bemade to the attached drawings. These drawings show different aspects ofthe present inventions and, where appropriate, reference numeralsillustrating like structures, components, materials and/or elements indifferent figures are labeled similarly. It is understood that variouscombinations of the structures, components, materials and/or elements,other than those specifically shown, are contemplated and are within thescope of the present inventions.

Moreover, there are many inventions described and illustrated herein.The present inventions are neither limited to any single aspect norembodiment thereof, nor to any combinations and/or permutations of suchaspects and/or embodiments. Moreover, each of the aspects of the presentinventions, and/or embodiments thereof, may be employed alone or incombination with one or more of the other aspects of the presentinventions and/or embodiments thereof. For the sake of brevity, many ofthose permutations and combinations will not be discussed and/orillustrated separately herein.

FIG. 1A is a block diagram representation of an exemplary light fielddata acquisition device;

FIG. 1B is a block diagram representation of an exemplary light fielddata acquisition device including, among other things, post-processingcircuitry integrated therein;

FIGS. 1C and 1F are block diagram representations of exemplary lightfield data acquisition systems including a light field data acquisitiondevice and post-processing circuitry;

FIG. 1D is a block diagram representation of an exemplary light fielddata acquisition device including memory (integrated therein) to storeLight Field Data;

FIG. 1E is a block diagram representation of an exemplary light fielddata acquisition device including, among other things, post-processingcircuitry and memory integrated therein;

FIG. 1G is a block diagram representation of an exemplary light fielddata acquisition device including a plurality of optics and sensors torecord, acquire, sample and/or capture light field data, includingmemory integrated therein;

FIG. 1H is a block diagram of an exemplary light field data acquisitiondevice including optics, a coded aperture, and sensor to record,acquire, sample and/or capture light field data, including memoryintegrated therein;

FIGS. 2A and 2B are block diagram representations of exemplaryrefocusable video acquisition units according to at least certainaspects of certain embodiments of the present inventions;

FIGS. 2C and 2E are block diagram representations of exemplaryrefocusable video acquisition units having removable components,including removable optics and light field or imaging recording units,according to at least certain aspects of certain embodiments of thepresent inventions, wherein in these embodiments, the acquisition unitincludes a microlens array in the optical path; notably, in theseexemplary embodiments the refocusable video acquisition unit may record,acquire, sample and/or capture light field or other (for example, 2D)video data;

FIGS. 2D and 2F are block diagram representations of exemplaryrefocusable video acquisition unit having removable components,including removable optics and microlens array, according to at leastcertain aspects of certain embodiments of the present inventions,wherein in these embodiments, the acquisition unit does not include amicrolens array in the optical path; notably, in this exemplaryembodiment the video acquisition unit may record, acquire, sample and/orcapture standard or conventional video data;

FIGS. 3A, 3C, 3D and 3F are block diagram representations of exemplaryrefocusable video acquisition systems having standalone refocusablevideo acquisition units (which may include on-system data storage)according to at least certain aspects of certain embodiments of thepresent inventions; with these embodiments, the refocusable videoacquisition system is not integrated the refocusable video editingsystems, but may couple to such systems as well as other externalsystems/devices (for example, external data storage, video display,recording device and/or data processing system);

FIGS. 3B, 3E and 3G are block diagram representations of exemplaryrefocusable video acquisition and editing systems having refocusablevideo acquisition systems integrated with the refocusable video editingsystems, according to at least certain aspects of certain embodiments ofthe present inventions; notably, the refocusable video acquisition andediting systems, in certain embodiments, couple to externalsystems/devices (for example, external storage, video display, recordingdevice and/or data storage);

FIG. 3H is a block diagram of an exemplary refocusable video editingsystem 100 containing refocusable video processing circuitry 102 andvideo processing circuitry 104, according to at least certain aspects ofcertain embodiments of the present inventions; notably, refocusablevideo, in certain embodiments, is processed first by refocusable videoprocessing circuitry and output video generated is then processed byvideo processing circuitry (for example, to apply “special effects” tothe output video);

FIG. 3I is a block diagram of an exemplary refocusable video editingsystem 100 containing refocusable video processing circuitry 102 and anexternal video processing system, according to at least certain aspectsof certain embodiments of the present inventions; notably, refocusablevideo, in certain embodiments, is processed first by refocusable videoprocessing circuitry and output video generated is then processedseparately by external video processing system (for example, to apply“special effects” to the output video);

FIG. 3J is a block diagram of an exemplary refocusable video editingsystem 100 containing refocusable video processing circuitry 102 andvideo processing circuitry 104, according to at least certain aspects ofcertain embodiments of the present inventions; notably, refocusablevideo, in certain embodiments, is processed first by video processingcircuitry (for example, to apply exposure adjustment) and refocusablevideo is then further processed by refocusable video processingcircuitry to generate output video;

FIG. 3K is a block diagram of an exemplary refocusable video editingsystem 100 containing refocusable video processing circuitry 102 andvideo processing circuitry 104, according to at least certain aspects ofcertain embodiments of the present inventions; notably, refocusablevideo, in certain embodiments, is processed first by video processingcircuitry (for example, to apply exposure adjustment) and refocusablevideo is then further processed by refocusable video processingcircuitry to generate output video, and output video is furtherprocessed by video processing circuitry (for example, to apply “specialeffects” to the video;

FIG. 4A is a block diagram representation of an exemplary refocusablevideo acquisition system according to at least certain aspects ofcertain embodiments of the present inventions wherein the refocusablevideo acquisition system of this exemplary embodiment includes a videoacquisition unit, lens optics, and a microlens relay adapter having amicrolens array disposed there between; notably, in this exemplaryembodiment, the microlens relay adapter includes an external lens mountto facilitate mechanically coupling the lens optics of the system;

FIGS. 4B-4E are block diagram representations of exemplary refocusablevideo acquisition system according to at least certain aspects ofcertain embodiments of the present inventions wherein the refocusablevideo acquisition system of these exemplary embodiments includereplaceable, substitutable and/or interchangeable modules wherein atleast one of the modules facilitates acquisition of refocusable videodata, for example, light field video data; notably, the light fieldmodule may include a microlens array, photo sensor, control circuitryand/or optics; moreover, the modules may be mechanically, electricallyand/or optically coupled to the base unit;

FIG. 5 is a block diagram representation of an exemplary user interfaceof, for example, refocusable video acquisition unit, according tocertain aspects of the present invention; notably, in one embodiment,the user interface may include an output device/mechanism (for example,display and/or speaker) and/or user input device/mechanism (for example,buttons, switches, touch screens, pointing device (for example, mouse ortrackball) and/or microphone) to allow a user/operator to monitor,control and/or program operating parameters of the refocusable videoacquisition unit (for example, (i) the rates of acquisition, sampling,capture and/or recording of refocusable data by the photo sensor and/or(ii) the focal plane, field of view or depth of field of the refocusablevideo acquisition unit);

FIG. 6A is a block diagram representation of an exemplary systemincluding a refocusable video acquisition unit, refocusable videoprocessing circuitry and an exemplary user interface including an outputdevice/mechanism (a display in this exemplary embodiment), according toat least certain aspects of certain embodiments of the presentinventions, wherein using the (“live”) video data acquired by therefocusable video acquisition unit, the refocusable video processingcircuitry may generate, determine, calculate and/or provide informationwhich is representative of those (i) ranges within the scene that may befocused or re-focused at one or more predetermined or given focalplane(s) and/or depths of field of refocusable video acquisition systemand/or (ii) subjects within the scene that may be within a predeterminedor given focus (after post-processing and based on a particular, givenor predetermined set of parameters of the refocusable video acquisitionunit); the refocusable video processing circuitry may provide suchinformation to the user interface (which may include a display orspeaker) thereby providing a guidance to the operator regarding theextent to which a processing system (for example, refocusable videoprocessing system of the present inventions) is able to compute, provideand/or generate one or more images having a particular or given focus ofthe subject(s) at one or more different depths in the acquired footage;

FIG. 6B is a block diagram representation of an exemplary systemincluding a refocusable video acquisition unit, refocusable videoprocessing circuitry and an exemplary user interface including an outputdevice/mechanism (a display in this exemplary embodiment), according toat least certain aspects of certain embodiments of the presentinventions, wherein using the current (“live”) video data acquired bythe refocusable video acquisition unit having an adjusted aperture, forexample, a small or reduced aperture relative to typical dataacquisition, the refocusable video processing circuitry may generate,determine, calculate and/or provide information regarding a subject,object and/or range at which the refocusable video processing circuitrymay refocus the video (for example, a maximum range where sharp focuscan be computed—after post-processing by the refocusable videoprocessing circuitry);

FIG. 7A is an image of a group of monks in different focal planeswherein the image focused on depth of closest eye in image; notably,this illustration is referenced in FIGS. 7B and 7C;

FIG. 7B is an image of a live computed depth map, generated for thescene presented in FIG. 7A, where the color or shades of the imageindicates the focal depth at which that portion of the image is“in-focus” (computed focus depth mask for image of FIG. 7A); in theillustration, darker colors or shades represent areas nearer toacquisition unit and brighter areas represent areas further from theacquisition unit;

FIG. 7C is an image of a computed, live, “in-focus” determination, usingthe image of FIG. 7A (computed in-focus or out-of-focus mask for theimage of FIG. 7A); notably, in the illustration, white represents areasdetermined to be “in-focus” and black represents areas determined to“out-of-focus”; intermediate colors, shades or values indicateintermediate proximity to “in-focus”;

FIG. 8A is an image of a computed (small aperture) view with a smallaperture, using the image presented in FIG. 7A.

FIG. 8B is an illustration indicating how a single pixel from or undereach microlens may be selected to create an image of a scene with alarge depth of field, for example, for “live” view; notably, the imageon the left is a zoomed and magnified section of a light field imagewith pixel near center of each projected microlens disk highlighted. Theblack box near the center of each disk is a representation of how analgorithm may select a single pixel near the center, and the lines andimage on the right show conceptually how the individual pixels from thedisk centers (in this exemplary embodiment) may be assembled into afinal image that may be employed, for example, in connection with “live”view;

FIG. 9 illustrates “screen-shots” of the display of the user interfaceof the refocusable video editing system associated with two frames and acorresponding timeline for such frames as generated, output and/orprovided by the refocusable video editing system, wherein the two framesmay be predetermined and/or key frames as set or defined within therefocusable video editing system according to at least certain aspectsof certain embodiments of the present inventions, wherein in a firstframe for which the parameters of focus and/or exposure are set ordefined in connection with the tree (right side of image) and a secondframe for which the parameters of focus and/or exposure are set ordefined in connection with the background outside (left side ofimage—outside the window); in this embodiment, the temporal position ofeach frame within the video data stream or output is indicated on atimeline below video window.

FIG. 10 illustrates a flow for the user or operator to select ordesignate a virtual focus parameter of one or more predetermined or keyframes wherein such virtual focus parameter may be employed by therefocusable video editing system to generate and output video datahaving such focus depth; in one embodiment, when rendering or generatingoutput video data, the virtual focus parameter may be defined ordetermined using key frames and interpolation between such key frames.The interpolation may be nearest neighbor interpolation, linearinterpolation, non-linear interpolation, cubic interpolation, Beziercurve interpolation, and/or spline-based interpolation; indeed, therefocusable video editing system may employ any interpolationtechnique(s) now known or later developed.

FIGS. 11A-11C illustrate exemplary embodiments of “Slow Focus Pull”according to at least certain aspects of certain embodiments of thepresent inventions, wherein the user or operator defines, designatesand/or inserts key frames having different focus depths and therefocusable video editing system implements a visual effect within theframes of the video data stream that provides an extended transition inthe focus depth from the time of the first defined (which was set to avirtual focus parameter of A), designated or set key frame to the seconddefined, designated or set key frame (which was set to a virtual focusparameter of B);

FIGS. 12A-12C illustrate exemplary embodiments of “Jump Focus Pull”according to at least certain aspects of certain embodiments of thepresent inventions, wherein the user or operator defines, designatesand/or inserts a first key frame which includes a focus depth parameterof A (which may be defined, designated and/or set by the user), at atime corresponding to a number of frames before the second key framewhich includes a focus depth parameter of B (again, which may bedefined, designated and/or set by the user), and the refocusable videoediting system implements a visual effect that includes a relativelyrapid or quick transition (for example, within 3-5 frames) from thefirst focus depth to the second focus depth;

FIGS. 13A-13C illustrate exemplary embodiments of “Peek Focus Pull”according to at least certain aspects of certain embodiments of thepresent inventions, wherein the user may define, designate and/or insertthree key frames as well as define and/or designate the focus depth ofeach key frame such that the first and third key frames may be set to afocus depth parameter of A, and the second key frame is set of a focusdepth parameter of B, and the refocusable video editing systemimplements a visual effect that includes a temporary transition from theoriginal focus depth, to the new focus depth and back to the originalfocus depth;

FIGS. 14A and 14B illustrate exemplary embodiments of “Focus SubjectTracking” according to at least certain aspects of certain embodimentsof the present inventions, wherein in response to user or operatorinputs, the refocusable video editing system may adjust, control and/ormanage focus on one or more subjects and adjust and track the virtualfocus parameter based thereon; notably, the virtual focus parameter ofthe designated frames may be user defined such that the system providesa focus depth of the designated subject(s) for a predetermined set offrames wherein such subject(s) is/are always in user predetermined ordefined focus (for example, the designated subject(s) in the refocusablevideo data is/are located in the focus plane);

FIG. 15 illustrates an exemplary embodiment of “Focus Subject Tracking”according to at least certain aspects of certain embodiments of thepresent inventions, wherein in response to user or operator inputs, therefocusable video editing system may adjust, control and/or manage thefocus on one or more subjects; notably, the virtual focus parameter ofthe designated frames may be user defined such that the system providesa focus depth of the designated subject(s) for a predetermined set offrames wherein such subject(s) is/are always in user predetermined ordefined focus (for example, the designated subject(s) in the refocusablevideo data is/are located in the focus plane);

FIG. 16 illustrates an exemplary embodiment of the focus depth analysisfor a predetermined amount of time of the video data or predeterminednumber of frames (for example, between key frames), according to atleast certain aspects of certain embodiments of the present inventions;

FIGS. 17A and 17B illustrate exemplary embodiments of exposurecorrection and/or adjustment based on subject or area analysis ofrefocusable video data, according to at least certain aspects of certainembodiments of the present inventions; in these exemplary embodiments,the refocusable video editing system may control, manage, adjust and/orchange the exposure parameter of a predetermined amount of refocusablevideo data (acquired by refocusable video acquisition system) so that,for example, the subject(s) of focus or area(s) of focus are exposedcorrectly and/or exposed in a predetermined manner;

FIG. 18 illustrates an exemplary embodiment of the flow of the aperturesize and/or shape adjustment and/or modification of the refocusablevideo data (acquired by refocusable video acquisition system), accordingto at least certain aspects of certain embodiments of the presentinventions; in these embodiments, the refocusable video editing systemmay modify and/or change the aperture size and/or shape parameters forvideo data associated with one or more frames of refocusable video data(acquired by refocusable video acquisition system 10) in accordance withsuch parameter(s); in the exemplary embodiment, refocusable videoediting system, based on the aperture size and/or aperture shapeparameter(s) of the key or predetermined frames, may interpolate betweenthe key or predetermined frames to determine the aperture size for alltimes in the video;

FIG. 19 illustrates a scene (viewing from overhead) having two subjectsin different focal planes; both subjects may be displayed “in-focus”wherein one of the focal planes is skewed or “tilted” (or off-axis)relative to the other focal plane (for example, the focal plane of theoptics of the refocusable video acquisition unit during acquisition,sampling and/or capture of the refocusable video data) wherein therefocusable video editing system may simulate a tilt-shift lens ingenerating output video using the refocusable video data;

FIG. 20 is a block diagram representation of an exemplary refocusablevideo acquisition and editing system having refocusable videoacquisition systems integrated with the refocusable video editingcircuitry, according to at least certain aspects of certain embodimentsof the present inventions; notably, the refocusable video acquisitionand editing system, in certain embodiments, couple to externalsystems/devices (for example, external storage, data processingcircuitry, video display, recording device and/or data storage);

FIG. 21A illustrates an exemplary embodiment of “Focus Planning”according to at least certain aspects of certain embodiments of thepresent inventions, wherein the user, refocusable video acquisitionsystem and/or refocusable video editing systems may evaluate or analyzea plurality of frames of the refocusable video data, including (i) datawhich is representative of video frames that precede the current and/ora given frame and (ii) data which is representative of video frame thatfollow the current and/or the given frame to analyze the video tocalculate, set, define, determine and/or obtain a predetermined, corrector desired focus depth;

FIG. 21B illustrates a block diagram representation of systems andtechniques to generate and output video data or frames (for example, toexternal storage, to a video display, to a recording device, and/or toprocessing circuitry) using one or more virtual focus parameters andrefocusable video data or frames which temporally corresponds to theoutput video data or frames, wherein one or more of the virtual focusparameters (which may include data which is representative of apredetermined, selected and/or desired (absolute or relative) virtualfocus depth) is/are determined using video data acquired after therefocusable video data or frames which temporally corresponds to theoutput video data or frames; notably, an exemplary timing diagram ofcertain operations of an exemplary embodiment is also illustrated inFIG. 21B;

FIG. 21C illustrates a timing diagram to generate and output video dataor frames (for example, using block diagram representation of system ofFIG. 21B) using one or more virtual focus parameters and refocusablevideo data or frames which temporally corresponds to the output videodata or frames, wherein processing circuitry may calculate, determineand/or generate a virtual focus characteristic or parameter inaccordance with, for example, the manner, form or type of virtual focustransition (similar to that as described herein in connection with keyframes), notably, the manner, form or type of focus transition maydefine or determine a virtual focus transition(s) (for example, the SlowFocus Pull, Jump Focus Pull, Peek Focus Pull, described herein (see,FIGS. 11A-13C)) from a particular virtual focus depth of a first virtualfocus parameter to a second virtual focus depth of a second virtualfocus parameter (in the illustrated embodiment, video framescorresponding to F_(t1)-F_(t2));

FIG. 21D illustrates an exemplary embodiment of “Focus Planning”according to at least certain aspects of certain embodiments of thepresent inventions, wherein the user, refocusable video acquisitionsystem and/or refocusable video editing systems may evaluate or analyzea plurality of frames of the refocusable video data, including (i) datawhich is representative of video frames that precede the frame beingrendered to output video, (ii) data which is representative of videoframe that follow the frame being rendered to output video, (iii) datawhich is representative of events that precede the frame being renderedto output video, and (iv) data which is representative of events thatfollow the frame being rendered to output video to analyze the video tocalculate, set, define, determine and/or obtain a predetermined, corrector desired focus depth;

FIG. 21E illustrates another exemplary embodiment of “Focus Planning”according to at least certain aspects of certain embodiments of thepresent inventions, wherein the refocusable video acquisition systemand/or refocusable video editing systems may evaluate or analyze forevents of interest refocusable video frames immediately or nearlyimmediately as they are acquired, and store a plurality of frames andevents of interest of the refocusable video data, including (i) datawhich is representative of video frame that follow the frame beingrendered to output video, (ii) data which is representative of eventsthat precede the frame being rendered to output video, and (iii) datawhich is representative of events that follow the frame being renderedto output video to analyze the video to calculate, set, define,determine and/or obtain a predetermined, correct or desired focus depth;

FIG. 22 illustrates an exemplary embodiment of the software autofocus todetermine a focus setting or parameter for the refocusable video data,according to at least certain aspects of certain embodiments of thepresent inventions; and

FIG. 23 illustrates an a block diagram representation of an exemplaryrefocusable video acquisition and editing system having on-system bufferand edit architecture, according to at least certain aspects of certainembodiments of the present inventions, in which the system includeson-system data storage, systems and techniques to determine videorendering parameters (for example, focus), systems and techniques forrendering video, and systems for saving and/or displaying rendered video(for example, to external storage, to a video display, to a recordingdevice, and/or to processing circuitry);

FIGS. 24A, 24B, 24C, and 24D are a set of four sub-aperture imagesgenerated from the same light field; wherein each sub-aperture image wasgenerated by sampling the light field using a different set of fixed Uand V coordinates including sub-aperture image for target U,Vcoordinates of −0.25, 0 (FIG. 24A), sub-aperture image for target U,Vcoordinates of 0.25, 0 (FIG. 24B), sub-aperture image for target U,Vcoordinates of 0, −0.25 (FIG. 24C), and sub-aperture image for targetU,V coordinates of 0.25, 0 (FIG. 24D);

FIG. 24E illustrates in a more conceptual manner how the sub-apertureimages presented in FIGS. 24A, 24B, 24C and 24D may be used to determinethe proper focus depth for objects in the scene by shifting andcomparison; the illustration shows the four sub-aperture imagescomposited and shifted relative to each other, in a manner consistentwith changing the focus of the image; in the composite, the four imagesof the glove in the foreground align, indicated that the glove is“in-focus” and the person is “out-of-focus”; that is, the four images ofthe person in the background are not aligned, and, in this embodiment,result in multiple “ghost” images, indicating the person is“out-of-focus”;

FIG. 25 is a sub-aperture representation generated by applying aLaplacian operator to a sub-aperture image; in some embodiments, thistype of sub-aperture representation may be used in lieu of sub-apertureimages for focus analysis;

FIG. 26 is an image illustrating split-screen focus; in the image, theleft side is focused on a focal plane that focuses on the image of thegirl on the left while the right half of the image is focused on a focalplane that focuses on the image of the girl on the right; and

FIGS. 27A and 27B are block diagrams that illustrate focus timelines attwo different and partially overlapping times during acquisition of therefocusable video data or frames.

Again, there are many inventions described and illustrated herein. Thepresent inventions are neither limited to any single aspect norembodiment thereof, nor to any combinations and/or permutations of suchaspects and/or embodiments. Each of the aspects of the presentinventions, and/or embodiments thereof, may be employed alone or incombination with one or more of the other aspects of the presentinventions and/or embodiments thereof. For the sake of brevity, many ofthose combinations and permutations are not discussed separately herein.

DETAILED DESCRIPTION

There are many inventions described and illustrated herein. In oneaspect, the present inventions are directed to refocusable videoacquisition devices/systems (hereinafter “device/system” and“devices/systems” are collectively “system” and “systems”, respectively)having refocusable video acquisition units that acquire, sample,capture, record and/or obtain refocusable video data (for example, videolight field data). In another aspect, the present inventions aredirected to refocusable video editing systems having video processingcircuitry that generates, manipulates and/or edits refocusable videodata (for example, light field data) which may be acquired, sampled,recorded and/or obtained via refocusable video acquisition systems.

The refocusable video acquisition systems of the present inventions mayemploy a rate of acquisition, sampling, capture, recording ofrefocusable image data (for example, light field data) which is higherthan “still” cameras and, is sufficiently high (for example, greaterthan 15 frames/sec, and preferably at least 24 frames/sec) toaccommodate providing or generating video quality output. Notably, inaddition to acquisition, sampling, capture, recording of refocusableimage data, the refocusable video acquisition systems may include, amongother things, on-system data storage to store refocusable video data.Indeed, the refocusable video editing systems may include, in additionto video processing circuitry, on-system data storage to storerefocusable video data processed thereby and/or obtained from therefocusable video acquisition system.

The present inventions, in certain aspects, are also directed to methodsor techniques for operating refocusable video acquisition systems and/orrefocusable video editing systems. Such methods or techniques may beimplemented on/in the refocusable video acquisition systems and/orrefocusable video editing systems of the present inventions.

With reference to FIG. 2, in one exemplary embodiment, refocusable videoacquisition system 10 includes refocusable video acquisition unit 12having optics 14, sensor 16, microlens array 18 and control circuitry20, to acquire, sample, record and/or obtain refocusable video data (forexample, light field video data). Here, microlens array 16 isincorporated into the optical path to facilitate acquisition, capture,sampling of, recording and/or obtaining light field video via sensor 18.The control circuitry 20 manages or controls (automatically or inresponse to user inputs) the rate of acquisition, sampling, capturing,recording and/or obtaining of refocusable data (for example, light fielddata). As such, where refocusable video acquisition system 10 isacquiring or generating video, control circuitry 20 provides for orimplements a sufficiently high rate of acquisition, sampling, capture,recording and/or obtaining of refocusable data (for example, greaterthan 15 frames/sec, and preferably at least 24 frames/sec) by sensor 18to facilitate generation or production of video quality output data. Therate of acquisition by sensor 18 may be fixed or programmable (forexample, via user/operator inputs).

Notably, where refocusable video acquisition system 10 is capable ofacquiring “still” images and in those instances where refocusable videoacquisition system 10 is programmed or configured to acquire or generate“still” images, control circuitry 20 may (in response, for example, touser/operator inputs) provide or implement lower rates of acquisition,sampling, capture, recording and/or obtaining of refocusable data (forexample, less than 3 frames/sec) by sensor 18. Again, the rate ofacquisition of refocusable image data by sensor 18 may be fixed orprogrammable.

The refocusable video acquisition system 10 may store the refocusablevideo data stream (for example, output by sensor 18) and/or therefocusable video data (for example, a representation of the refocusablevideo data stream) in external system data storage 22 and/or inon-system data storage 24. (See, for example, FIGS. 3A-3F). In thisregard, refocusable video acquisition system 10 may store the “raw”refocusable video data stream (as output from sensor 18) and/or arepresentation thereof (for example, a compressed refocusable video datastream, multiple video data stream (for example, a plurality (3-10) ofvideo data stream of the same scene, each data stream being refocused atdifferent focal planes) and/or combinations thereof). All permutationand combinations of data storage formats of the refocusable video datastream and/or a representation thereof are intended to fall within thescope of the present inventions. Notably, refocusable video data streamand/or representations thereof may be collectively referred to as“refocusable video data”).

In one exemplary embodiment, refocusable video acquisition system 10 mayemploy the optics, microlens array and/or sensor of one or more of thelight field data acquisition units of FIGS. 1A-1F. The light field dataacquisition discussions set forth in United States Patent ApplicationPublication 2007/0252074, the provisional applications to which itclaims priority (namely, Ser. Nos. 60/615,179 and 60/647,492), and RenNg's PhD dissertation, “Digital Light Field Photography”) areincorporated herein by reference. In other exemplary embodiments,refocusable video acquisition system 10 employs certain aspects of suchlight field acquisition devices (for example, the optics and microlensarray) in conjunction with other components, circuitry and/or elements(for example, a high speed photo sensor).

Notably, refocusable video acquisition system 10 of the presentinventions may be a stand-alone system/device (see, FIGS. 3A, 3C, 3D and3F) or may be integrated with a refocusable video editing system (see,FIGS. 3B, 3E and 3G). That is, refocusable video acquisition system 10may be separate from a refocusable video editing system (for example,refocusable video editing system 100 according to the presentinventions) which receives acquires and/or obtains refocusable videodata, information, images and/or frames therefrom and performs dataprocessing thereon, and, in other exemplary embodiments, refocusablevideo acquisition system 10 may be integrated (or substantiallyintegrated) with a refocusable video editing system (see, for example,FIGS. 3A, 3C, 3D and 3F and refocusable video editing system 100according to the present inventions).

The refocusable video acquisition unit 12 may include fixed or permanentcomponents (for example, the optics, sensor, microlens array) orremovable components (for example, a removable microlens array). Inthose embodiments where the components of the refocusable videoacquisition unit are fixed or permanent, the optics, sensor and/ormicrolens array may be responsively or automatically adjusted (forexample, in a relative manner) to change, for example, the focal plane,field of view or depth of field of refocusable video acquisition unit12. In those embodiments, where refocusable video acquisition system 10includes one or more removable components, such components may becontrollably, temporarily and/or selectively incorporated into orremoved from refocusable video acquisition unit 12. For example, withreference to FIGS. 2C-2F, refocusable video acquisition unit 12 mayinclude removable microlens array and/or unit containing a microlensarray and sensor 16 wherein, in one position (see, FIGS. 2C and 2E),microlens array 16 is located or disposed in the optical path tofacilitate acquisition, sampling and/or recording of refocusable videodata (for example, light field video data), and, in another position(see, FIGS. 2D and 2F), the microlens array is removed from the opticalpath of refocusable video acquisition unit 12 (and thereby removed oreffectively removed from refocusable video acquisition unit 12). In thisway, in addition to sampling, acquiring, sensing, capturing and/orrecording data which is representative of the light field (i.e., whenthe microlens array is in the optical path), refocusable videoacquisition system 10 may also acquire, capture and/or record standardor conventional video data when the microlens array is not located ordisposed within the optical path of system 10.

In another exemplary embodiment, the refocusable video acquisitionsystem may include a microlens relay adapter to couple or be positionedin the optical path, between the imaging sensor and optics of therefocusable video acquisition system, to project the image or scene onthe photo sensor. For example, with reference to FIG. 4A, microlensrelay adapter 26, in this embodiment, includes relay optics 28 andmicrolens array 16. The relay optics 28 may be positioned, disposed,located and/or situated between microlens array 16 and imaging sensor 18of the video acquisition device, such that relay optics 28 images (orfacilitates imaging) the focal plane of microlens array 16 onto imagingsensor 18. In one embodiment, relay optics 28 may include one or morelens elements, windows, apertures and mirrors. As such, in one exemplaryembodiment, relay optics 28 may provide a selected, predetermined orprogrammable (for example, user or operator programmable) magnification(for example, unit magnification).

Notably, it may be advantageous to design or provide the relay opticswith an f-number that is lower than the f-number of the microlens array,and with a diameter that is as large as or exceeds the width, height anddiagonal length of the microlens array. Such a design may reduce,minimize and/or eliminate vignetting of light rays diverging from theperiphery of the microlens array at external angles away from theoptical axis.

With continued reference to FIG. 4A, in one exemplary embodiment, theexterior end of microlens relay adapter 26 may include a mechanicalinterface providing an interchangeable lens mount. In this way,multiple/different external photographic lenses (generally illustratedas “lens optics 30”) may be mechanically mounted to refocusable videoacquisition system 10 in a manner similar to conventional cameraarchitectures.

In yet another exemplary embodiment, refocusable video acquisitionsystem 10 includes a plurality of replaceable, substitutable and/orinterchangeable acquisition modules wherein at least one of the modulesfacilitates acquisition of refocusable video data, for example, lightfield video data. For example, with reference to FIG. 4B, in oneembodiment, refocusable video acquisition system 10 includes light field(i) module 32 having a microlens array and/or optics to facilitateacquisition, sampling, capturing and/or recording of light field videodata, and (ii) non-light field type module 34 having elements, forexample, optics (i.e., without a microlens array and related/associatedoptics), to facilitate acquisition, sampling, capturing and/or recordingof non-light field video data (for example, standard 2D video data).Accordingly, in this embodiment, when light field module 32 ismechanically or optically coupled to refocusable video acquisitionsystem 10, refocusable video acquisition system 10 acquires, samples,captures, records and/or obtains light field video data which may,thereafter be provide to a refocusable video editing system whichgenerates, manipulates and/or edits refocusable video data. When,however, non-light field module 34 is mechanically or optically coupledto base unit 36, the video acquisition system 10 acquires, samples,captures, records and/or obtains non-light field video data (forexample, conventional or standard video data).

With reference to FIG. 4C, in another embodiment, light field module 32includes a light field sensor, microlens array and optics to acquire,sample, capture, record and/or obtain light field video data, and asecond module may contain a non-light-field sensor and/or optics (i.e.,without a microlens array and related/associated optics), to capturenon-light field video data (for example, standard 2D video data). Thus,in this embodiment, an appropriate or suitable photo sensor is disposedor provided within modules 32 and 34. As such, when light field module32 is mechanically, electrically and/or optically coupled to base unit36, video acquisition system 10 acquires, samples, captures, recordsand/or obtains light field video data. Indeed, when light field module32 of this embodiment is coupled to base unit 36, such video acquisitionsystem is a functioning as a refocusable video acquisition system. When,however, non-light field module 34 is mechanically, electrically and/oroptically coupled to base unit 36, the video acquisition system does notfunction or operate as a refocusable video acquisition system but rathermay acquire, sample, capture, record and/or obtain conventional orstandard video data.

Notably, in this embodiment, base unit 36 may be a platform to receiveone of the modules. The base unit 36 may include, for example, userinterface 38 (for example, a display and a user input device/mechanism)as well as control circuitry 20 to facilitate acquisition or capture ofvideo data, including refocusable video data.

With continued reference to FIGS. 4B and 4C, control circuitry 20 may beconfigured to detect or determine (automatically and/or in response touser inputs) the type of module attached or coupled to the acquisitionor base unit. In response thereto, control circuitry 20 may configurethe operation of refocusable video acquisition system 10 accordingly.That is, when light field module 32 is mechanically, electrically and/oroptically coupled to the video acquisition unit or base unit, controlcircuitry 20 configures system 10 to acquire, sample, capture, recordand/or obtain light field video data. Similarly, when non-light fieldmodule 34 is mechanically, electrically and/or optically coupled to thevideo acquisition unit or base unit, control circuitry 20 configuressystem 10 to function or operate as a refocusable video acquisitionsystem but rather may acquire, sample, capture, record and/or obtainconventional or standard video data.

Notably, control circuitry 20 may detect or determine the type of modulebased on any technique, circuitry and/or architecture (for example,physical/mechanical architecture) now known or later developed. Forexample, control circuitry 20 may automatically detect or determine amechanical, optical and/or electrical “signature” or “key” of themodule. In this regard, each module may have a different “signature” or“key” when coupled to the video acquisition unit—for example, the modulemay include a non-volatile memory to store data which is representativeof type of module; in this way, when the module is coupled to theacquisition unit, control circuitry 20 may detect or determine themodule type based on the data stored in the memory.

Notably, in another set of embodiments, control circuitry 20 is disposedin light field module 32. (See, FIGS. 4D and 4E). In these embodiments,control circuitry 20 coordinates, controls, configures and/or manages(automatically or in response to user inputs) the rate of acquisition,sampling, capture, recording and/or obtaining of refocusable video data(for example, light field data) when light field module 32 ismechanically, electrically and/or optically coupled to base unit 36.Moreover, as discussed in detail below, control circuitry 20 maycoordinate, control, configure and/or manage (automatically or inresponse to user inputs) other aspects or parameters of the acquisition,sampling, capture, recording and/or obtaining of refocusable video dataincluding, for example, the focal plane, field of view or depth of fieldof refocusable video acquisition unit 12 during acquisition, sampling,capture, recording and/or obtaining of refocusable video data.

Live “Is Shot Refocusable?”

As noted above, the refocusable video acquisition unit may include auser interface to allow a user/operator to monitor, control and/or theprogram refocusable video acquisition unit and/or refocusable videoacquisition system. For example, with reference to FIG. 2B, in oneembodiment, user interface 38 may include an output device/mechanism(for example, display and/or speaker) and/or user input device/mechanism(for example, buttons, switches, touch screen, pointing device (forexample, mouse or trackball) and/or microphone) to allow a user/operatorto monitor, control and/or the program operating parameters ofrefocusable video acquisition unit 12 (for example, (i) the rates ofacquisition, sampling, capture and/or recording of refocusable data bysensor 18 and/or (ii) the focal plane, field of view or depth of fieldof refocusable video acquisition unit 12). (See, FIG. 5).

In one embodiment, an indicator (visual or audible) may be disposed on,in or near the refocusable video acquisition system to provideinformation to the operator or user about the extent to which theprocessing system (for example, refocusable video processing circuitryof the present inventions) is able to generate, compute and/or provideone or more predetermined focused images (for example, sharply focused)of one or more subjects at one or more different focal plane(s), fieldof view(s) and/or depths of field of refocusable video acquisitionsystem (for example, a system having a refocusable video acquisitionunit 12 of FIG. 2B) in the acquired, to be acquired, and/or shotfootage. For example, with reference to FIGS. 2B, 5 and 6, refocusablevideo acquisition system 10 may include user interface 38 having adisplay providing a view to the operator that provides informationregarding the extent to which a processing system (for example,refocusable video processing system of the present inventions) is ableto generate, compute and/or provide one or more images having aparticular or given focus of the subject(s) at one or more differentdepths in the acquired footage. An exemplary indicator is illustrated inFIG. 7C wherein, relative to FIG. 7A, white portions represent areasdetermined to be “in-focus” and black portions represent areasdetermined to “out-of-focus”; portions have intermediate colors, shadesor values indicate intermediate proximity to “in-focus”.

Notably, user interface 38 and/or the refocusable video processingcircuitry may be integrated, disposed and/or located on/in refocusablevideo acquisition system 10 or unit 12.

In this embodiment, the refocusable video processing circuitry maygenerate, determine, calculate and/or provide information which isrepresentative of those ranges within the scene that may be focused orre-focused at one or more predetermined or given focal plane(s) and/ordepths of field of refocusable video acquisition system (afterpost-processing and based on a particular, given or predetermined set ofparameters of the refocusable video acquisition unit). In additionthereto, or in lieu thereof, the refocusable video processing circuitrymay generate, determine, calculate and/or provide information which isrepresentative of those subjects within the scene that may be within apredetermined or given focus (after post-processing and based on aparticular, given or predetermined set of parameters of the refocusablevideo acquisition unit). The refocusable video processing circuitry mayprovide such information to the user interface (which may include adisplay or speaker) thereby providing information or guidance to theoperator regarding the extent to which a processing system (for example,refocusable video processing system of the present inventions) is ableto generate, compute, and/or provide one or more images having apredetermined, particular or given focus of the subject(s) at one ormore different depths in the acquired footage.

Notably, the refocusable video processing circuitry may be implementedvia a plurality of discrete or integrated logic, and/or one or morestate machines, special or general purpose processors (suitablyprogrammed) and/or field programmable gate arrays (or combinationsthereof). Indeed, all circuitry (for example, discrete or integratedlogic, state machine(s), special or general purpose processor(s)(suitably programmed) and/or field programmable gate array(s) (orcombinations thereof)) to generate, determine, calculate and/or provideinformation which is representative of (i) those ranges within the scenethat may be focused at one or more predetermined or given focal plane(s)and/or depths of the field of refocusable video acquisition system(after post-processing and based on a given or predetermined set ofparameters of the refocusable video acquisition unit) and/or (ii) thosesubjects within the scene that may be within a predetermined or givenfocus (after post-processing and based on a given or predetermined setof parameters of the refocusable video acquisition unit), consistentwith inventions described and/or illustrated herein, are intended tofall within the scope of the present inventions.

The following exemplary embodiments illustrate various methods forpresenting such information. However, it should be understood that thepresent inventions are intended to cover or encompass allrepresentations or views that provide the operator with informationwhich is representative of the extent of the focusing the refocusablevideo processing circuitry may provide in connection with refocusablevideo data which is to be acquired or has been acquired by therefocusable video acquisition unit based on a given or predetermined setof parameters.

Live Computed Aperture Adjustment

In one exemplary embodiment, using the current (“live”) video dataacquired by the refocusable video acquisition unit, the refocusablevideo processing circuitry may generate, determine, calculate and/orprovide information regarding a subject, object and/or range at whichthe refocusable video processing circuitry may refocus the video (forexample, a maximum range where sharp focus can be determined and/orcomputed—after post-processing by the refocusable video processingcircuitry). The refocusable video processing circuitry may generate,determine, calculate and/or provide information regarding a subject,object and/or range at which the refocusable video processing circuitrymay refocus the video using data acquired by the refocusable videoacquisition unit having an adjusted aperture, for example, a small orreduced aperture relative to typical data acquisition. In thisembodiment, the refocusable video processing circuitry may generateinformation which is representative of a current or “live”, computedview using an adjusted aperture (for example, a small or reducedaperture). Such information may be presented to the user or operator ofthe scene being acquired or shot by a refocusable video acquisitionunit—for example, a current or live view that contains a depth of fieldwhich is larger than the optical depth of field. The size of thecomputed aperture may be selected such that the objects may appear less“sharp” in the operator's view (for example, of the user interface) asthe subject, object and/or range reaches, for example, a predetermined,particular and/or given focus that may be computed, generated and/orprovided the refocusable video processing circuitry (for example, amaximum range where sharp focus can be computed—after post-processing bythe refocusable video processing circuitry). This exemplary embodimentis shown, in block diagram form, in FIG. 6B.

In one exemplary embodiment, refocusable video processing circuitry 102may generate, compute and/or output video data or still image data whichfacilitates providing such a view (for example, on a display of the userinterface 38) by synthesizing a view with an aperture that isapproximately 1/N of the diameter of the optical aperture, where N isthe number of pixels under a microlens disk of the microlens array ofthe refocusable video acquisition unit 12 in the raw light field imageacquired or obtained thereby. (See, for example, FIG. 8A).

Notably, any method for computing an extended depth of field image,whether now known or later developed, may be used and is intended tofall within the scope of the present inventions. For example, in oneexemplary embodiment, the center-most pixel in the micro image thatforms under each microlens (referred to in the sentences below as a“microlens image”) of the microlens array may be combined with all othersuch pixels in an array in order to compute the small-aperture view withextended depth of field appearance (See, for example, FIG. 8B). Inanother embodiment, the center-most pixel value is substituted with avalue interpolated from a set of pixels in a neighborhood closest to thecenter of the microlens image. In one exemplary embodiment, this set ofpixels comprises the four pixels closest to the center of the microlensimage, and the interpolation function is used is bilinear interpolationaccording to the distance of the four pixel centers from the geometriccenter of the microlens image.

Live Depth Map Display

In addition thereto, or in lieu thereof, in another exemplaryembodiment, the refocusable video processing circuitry may generate,compute and/or provide information which is representative of a “live”,computed focus depth “map” of the scene (which may be based on theoperating or acquisition parameters of the refocusable video acquisitionunit). The map of the focus depth calculated, determined and/orgenerated by the refocusable video processing circuitry may be displayedto the user/operator in real time or near real-time. The view may beshown in various manners on the display of the user interface—forexample, as an overlay on the video stream of the video data, or as aninset video stream. (See, for example, FIG. 7B and FIG. 7C).

In one embodiment, the appearance of the depth map at different regionsof the field of view may indicate the focus deviation in that regionfrom the optical focus point. The appearance may, for example, vary incolor (for example, different colors or a scale of a given color (forexample, gray scale)) and/or brightness according to the deviation fromthe focus point. In another embodiment, the map of the focus depth mayinclude absolute or relative values such that the user/operator maydetermine the capability or potential of adjusting or changing the focusof a particular subject, region or range in the scene. Any manner ortechnique to provide such information to the user/operator, whether nowknown or later developed, is intended to fall within the scope of thepresent invention.

The focus depth map may be determined or estimated using any method nowknown or later developed. For example, the map may be computed andutilized as described in the following exemplary embodiments:

-   -   From the current light field frame captured by the system,        generate, compute and/or determine a set of images refocused at        a set of depths that includes depths closer and further than the        depths that may be focused at the desired or predetermined level        of sharpness.    -   For each region in the image, generate, compute and/or determine        the image that contains predetermined details (for example, the        sharpest details) in that portion of the image. One method for        doing so applies a focus metric, such as an edge detection image        processing algorithm to the images, and determines the image in        which the focus metric is highest in the region of interest.    -   Each region is conceptually tagged with the focus depth of the        image containing the predetermined details (for example, the        sharpest details). If the focus depth corresponds to a depth        outside the range of a predetermined depth that allows a        predetermined focus at the desired or a predetermined level of        sharpness, the region is conceptually tagged with an “out of        range” marker.    -   In embodiments where the appearance is set according to the        focus depth, the focus depth label is used. In another exemplary        embodiment regions marked with an “out of focus” indicator are        presented with a blinking appearance in the view in order to        draw the user's attention to that problem area.        “Out of Refocusable Range Icon”

In addition thereto, or in lieu thereof, in one exemplary embodiment,the refocusable video processing circuitry may generate, compute and/orprovide information which is representative of an “out of refocusablerange” information (for example, an icon that may be provided to adisplay on the user interface). In this regard, in one embodiment, anicon representative of an “out of refocusable range” (for example, asdetermined by refocusable video processing circuitry using “live” orcurrent video data) may be displayed on the user interface by controlcircuitry when the scene contains subjects or items outside the range ofdepths that the refocusable video processing circuitry may refocus (forexample, refocus sharply). Notably, any method now known or laterdeveloped may be used to determine or estimate whether the scenecontains such subjects. One exemplary method is as follows:

-   -   From the current light field frame captured by the system,        generate, compute and/or determine a set of images refocused at        a set of depths that includes depths closer and further than the        depths that may be focused at the desired level of sharpness.    -   For each region in the image, compute and/or identify the image        that contains the sharpest details in that portion of the image.        One method for doing so applies a focus metric, such as an edge        detection image processing algorithm to the images, and        determines the image in which the focus metric is highest in the        region of interest. As such, in one embodiment, the refocusable        video processing circuitry implements an edge detection        technique to identify the image that contains the sharpest        details in that portion of the image.    -   If any region of the image is sharpest in an image corresponding        to a depth outside the range of depths that can be refocused        sharply, the refocusable video processing circuitry generates        “out of refocusable range” information which may be displayed on        the user interfaces as an “out of refocusable range icon.        Another exemplary method, using sub-aperture images, is as        follows:    -   From the current light field frame captured by the system,        determine a set of sub-aperture images to be generated and/or        computed, where each sub-aperture image is a view of the light        field from a small portion of the aperture. In one embodiment,        these sub-aperture images can be identified by target U,V        coordinates in the X,Y,U,V light field coordinate system. In        this embodiment, each sub-aperture image 2 dimensional sampling        of the light field where U and V are held constant or near        constant. In an exemplary embodiment, the set of sub-aperture        images to be generated and/or computed is determined by a        regular sampling of the U,V coordinate space (for example,        sampling on a square or hexagonal grid).    -   From the current light field frame captured by the system,        generate and/or compute the determined set of sub-aperture        images or representations, where each sub-aperture image or        representation is a view of the light field or specific light        field characteristics from a small portion of the aperture. In        some embodiments, a sub-aperture image or representation is an        image created by sampling the current light field frame. In one        embodiment, a sub-aperture image may be generated for a U,V        value by sampling each microlens and returning the single pixel        value with U,V coordinates closest to the target U,V        coordinates. In another embodiment, the pixels under each        microlens are sampled and return a pixel value from bilinear        interpolation of the four pixels with nearest U,V coordinates to        the target U,V coordinates. (See FIG. 24A, FIG. 24B, FIG. 24C        and FIG. 24D). Indeed, any interpolation technique(s) now known        or later developed may be used, including but not limited to:        nearest neighbor interpolation, linear interpolation, non-linear        interpolation, cubic interpolation, Bezier curve interpolation,        and/or spline-based interpolation. In some embodiments, a        sub-aperture image or representation may be obtained by applying        a function to another sub-aperture image or representation. In        one specific exemplary embodiment, a sub-aperture image or        representation may be generated by applying an N by M or N by N        (for example, 3 by 3, 5 by 5, or 7 by 7) Laplacian operator to a        previously generated sub-aperture image or representation (see        FIG. 25). In another specific exemplary embodiment, a        sub-aperture image or representation may be generated by        applying a derivative operator (for example, sum modified        Laplacian or gradient magnitude) to a previously generated        sub-aperture image or representation. In other embodiments, a        sub-aperture image or representation may be generated by        applying an edge detection algorithm (for example, Canny edge        detection) to a previously generated sub-aperture image or        representation.    -   For each region of the image, compute and/or determine        similarity, consistency and/or difference scores. In one        embodiment, for a series of focus depths, each sub-aperture        image or representation is shifted relative to the other        sub-aperture images or representation and then compared for        regional image similarity and/or consistency. In one exemplary        embodiment, the sub-aperture images or representations are        shifted relative to each other according to the following        formula:        Dx=U*Focus_Depth        Dy=V*Focus_Depth    -   In one embodiment, the shifted sub-aperture images or        representations are compared for regional similarity or        consistency by applying a localized cross-correlation across all        sub-aperture images for each image region (See FIG. 24E). In an        exemplary embodiment, for each location, an N by N (for example,        11 by 11) pixel cross-correlation window is used, centered at        the target location. In this embodiment, sampling of the        sub-aperture images can be nearest neighbor, bilinear, or any        other sampling image sampling method now known or later        developed. In another embodiment, the shifted sub-aperture        images or representations are compared for regional similarity,        consistency and/or differences by applying localized statistical        metrics across all sub-aperture images for each image region. In        a specific exemplary embodiment, each shifted sub-aperture image        or representation is sampled at each location, and the set of        sample values at each location are statistically analyzed to        determine a score representing the similarity, consistency        and/or difference of the sample values at that location. In one        embodiment, the metric is the statistical variance of the        values. In another embodiment, it is the standard deviation of        the values. In another embodiment, the metric is the square root        of the standard deviation of the sample values. Indeed, any        statistical metric now known or later developed by be used to        calculate, generate or determine a similarity, consistency        and/or difference score for a location. In some embodiments, the        similarity, consistency and/or differences may be locally or        spatially combined or aggregated. In a specific exemplary        embodiment, the similarity, consistency and/or difference scores        are summed together over an N by N (for example, 9 by 9) region        to create an aggregate value.    -   For each region of the image, compute and/or determine the depth        or depths of sharpest focus based using the similarity,        consistency and/or difference scores. In some embodiments, the        similarity, consistency and/or difference scores may be used        directly to determine a localized depth of sharpest focus by        choosing the focus depth corresponding most “in-focus” scores        (for example, by choosing the largest value for scores        representing consistency or similarity (for example, cross        correlation scores) or the smallest values for scores        representing differences (for example, variance or standard        deviation scores). In other embodiments, the similarity,        consistency and/or difference scores may be aggregated, combined        and/or processed to determine the focus depth that is most        “in-focus”. In one specific exemplary embodiment, the        similarity, consistency and/or difference scores are combined        over an N by N region (for example, 5 by 5) and then selecting        the most “in-focus” depth based on the aggregate score. In        another exemplary embodiment, similarity, consistency and/or        difference scores are aggregated until a minimum total score or        confidence level is reached for the region (for example,        aggregation is performed until the total score is 10 times the        maximum score for any single location), and then selecting the        most “in-focus” depth based on the aggregate score.    -   If any region of the image is sharpest in an image corresponding        to a depth outside the range of depths that can be refocused        sharply, the refocusable video processing circuitry generates        “out of refocusable range” information which may be displayed on        the user interfaces as an “out of refocusable range icon.

Notably, in addition to or in lieu of the visual indicator, therefocusable video data acquisition system may include an audibleindicator (for example, a microphone and/or speaker) to provideinformation to the operator or user about the extent to which therefocusable video processing circuitry is able to generate and/orcompute predetermined (for example, sharply) focused images of thesubjects at different depths in the acquired footage. In addition, theoperator or user may identify, for example, the subject, via verbalinput/commands. As such, the discussion above pertaining to the visualindicator is fully applicable to an audible indicator wherein theinformation is provided to the operator or user via sound. For the sakeof brevity, that discussion will not be repeated.

Storing/Saving

In those embodiments where the refocusable video data is video lightfield data, such data may be characterized or interpreted as a 5Dfunction, with each pixel assigned a 5-tuple (t,x,y,u,v) representing aray in 3D-space at a particular time. In the 5-tuple: t is the time(video frame), (x,y) are the spatial coordinates of the raycorresponding to which microlens the pixel is under, and (u,v) are thedirectional coordinates of the ray corresponding to its position withinthe image that appears under each microlens.

The refocusable video acquisition system 10 and/or refocusable videoediting system 100 may store refocusable video data (for example, videolight field data) or data that is generated or derived therefrom, may bestored or saved in any format, including those exemplary embodimentsdescribed below. It should be noted, that the refocusable video data(for example, video light field data) or data that is generated orderived therefrom may be saved in any format, whether now known or laterdeveloped, that is consistent with the inventions set forth herein,including, for example:

-   -   Full raw light field video stream (uncompressed data)        -   All pixels (for example, the individual sensor elements of a            sensor array) in the (t,x,y,u,v) space are stored as they            are read off the sensor.    -   Compressed video light field        -   The refocusable video acquisition system 10 and/or            refocusable video editing system 100 may implement any            compression method now known or later developed to compress            the 5D (t,x,y,u,v) data set. The following is intended as an            illustrative, but not limiting, set of exemplary embodiments            of the compression component of the following invention:            -   The 5D data-set may be broken into contiguous blocks in                time of a set of light field video frames. These blocks                are interpreted as a 5D function, and compressed with a                5D wavelet compression technique.            -   Each frame is interpreted as a 4D light field in                (x,y,u,v), and it is compressed using a 4D wavelet                compression scheme.            -   Each frame of the video is interpreted as a 2D image,                ignoring its 4D light field structure, and compressed                using a 2D image compression technique, such as 2D                wavelets, or JPEG.            -   The video is interpreted as a regular (t,x,y) video                stream, ignoring the directional (u,v) structure of the                light field. It is compressed using regular video                compression techniques, including 3D wavelets, motion                JPEG, H.264, MPEG-4, etc.    -   Complete refocus stack (4D compression), comprising a set of        images focused at various focus depths for each time frame        -   In these exemplary embodiments, refocusable video            acquisition system 10 and/or refocusable video editing            system 100 may process each acquired or captured frame of            the light field data to generate, compute and/or output a            set of images focused at various depths, and these images            form the refocus stack that is saved for the current frame.            The refocusable video acquisition system 10 and/or the            refocusable video editing system 100 may employ any            refocusing technique now known or later developed.    -   Track top N potential subjects (N×compressed video stream)        -   In this exemplary embodiment, refocusable video acquisition            system 10 and/or refocusable video editing system 100 may            process each acquired or captured frame of the light field            to generate, compute and/or output a set of N images focused            on the N most important subjects, and these N images are            stored for that frame.            In particular, in one exemplary embodiment, for each frame,            refocusable video editing system 100 generates and/or            computes a set of images at a range or a plurality of focus            depths. The refocusable video acquisition system 10 and/or            refocusable video editing system 100 may analyze these            images to identify predetermined regions (for example, sharp            regions), utilizing, for example, one or more of the            techniques described herein. Each image receives a “score”            corresponding to the total size of the predetermined regions            (for example, sharp regions). The N images with the highest            score are chosen for the frame being considered.    -   In an exemplary embodiment, N is 2; in another embodiment, N is        1≧N≧5.    -   In another exemplary embodiment, the score assigned to each        image is higher if its sharp regions correspond to a human face.        A face detection algorithm is applied to determine or compute        which portions of each image, if any, correspond to a human        face.    -   Software autofocus (one compressed video stream)        -   In these exemplary embodiments, refocusable video            acquisition system 10 and/or refocusable video editing            system 100 may process the light field for each frame of the            video in order to generate, compute and/or provide a one            image that is saved for that frame, where the image is            automatically focused on a subject of interest.

In those situations where full focus selection is available inpost-production (for example, in the case of a professional typesystem), the refocusable video acquisition system 10, in one embodiment,may save/store the full or entire light field video stream, in eithercompressed or uncompressed forms, or a sufficient amount of theinformation which is representative of the light field video data streamor pertaining to the light field video data stream as necessary toimplement such post-processing thereon (by, for example, refocusablevideo editing system 100).

In the preceding exemplary embodiments, various computations aredescribed. These computations may be implemented with processingcircuitry disposed (in part or in whole) in/on the refocusable videoacquisition system 10 or in/on an external processing system (forexample, refocusable video editing system 100). The integrated circuitrymay include one or more microprocessors, Application-Specific IntegratedCircuits (ASICs), digital signal processors (DSPs), and/or programmablegate arrays (for example, field-programmable gate arrays (FPGAs)).Indeed, the processing circuitry may be any type or form of circuitrywhether now known or later developed. For example, the processingcircuitry may include a single component or a multiplicity of components(microprocessors, ASICs and DSPs), either active and/or passive, whichare coupled together to implement, provide and/or perform a desiredoperation/function/application.

Further, as mentioned above, in operation, the processing circuitry mayperform or execute one or more applications, routines, programs and/ordata structures that implement particular methods, tasks or operationsdescribed and illustrated herein. The functionality of the applications,routines or programs may be combined or distributed. Further, theapplications, routines or programs may be implemented by the sensor dataprocessing circuitry using any programming language whether now known orlater developed, including, for example, assembly, FORTRAN, C, C++, andBASIC, whether compiled or uncompiled code; all of which are intended tofall within the scope of the present invention.

Editing of Refocusable Video Data

This section describes exemplary embodiments relating to a refocusablevideo editing system 100 and techniques employed therein or thereby inconnection with refocusable video data and/or the operation or controlof refocusable video acquisition system 10. The exemplary embodimentspertaining to refocusable video editing system and technique employedthereby are at times described in the context where refocusable videodata includes light field video data. However, it is intended that theseexemplary embodiments, although set forth in the context of light fieldvideo data, are applicable to and may utilize any refocusable videodata, whether now known or later developed, where that video dataenables computation of images focused at different depths for variousframes of the video.

Briefly, with reference to FIGS. 3A-3G, refocusable video editing system100 includes refocusable video processing circuitry 102 to process,evaluate and/or analyze refocusable video data and to generate, computeand/or output video data, for example, as discussed herein. In oneembodiment, such data may be final video output and/or data whichfacilitates control and/or operation of refocusable video acquisitionsystem 10 (for example, refocusable video processing circuitry 102 maygenerate, compute and/or provide information which is representative ofan “out of refocusable range” information (for example, an icon that maybe provided to a display on the user interface). As noted above, the“out of refocusable range” “icon” may be displayed on the user interfaceso that the user or operator may control the operation of refocusablevideo acquisition system 10.

Notably, refocusable video editing system 100 may implement or couple tocircuitry that implements conventional video processing (for example,incorporation of “special effects”, color adjustments, and/or“sharpening” the video frames). For example, with reference to FIGS.3H-3K, refocusable video editing system 100 may include and/or couple tovideo processing circuitry 104 and/or external video processing system106. In this embodiment, the video processing implemented in conjunctionwith and/or appropriate to conventional video or video data may beimplemented or applied before and/or after processing of refocusablevideo data by refocusable video processing circuitry, for example,processing which converts the refocusable video data to output videodata and/or generating or rendering output video data using refocusablevideo data. In one embodiment, with reference to FIG. 3H, refocusablevideo editing system 100 may include refocusable video processingcircuitry 102 and video processing circuitry 104. In this embodiment,refocusable video processing circuitry receives as input refocusablevideo data and renders “intermediate” output video (for example, 2Dvideo). Thereafter the intermediate video may be processed by videoprocessing circuitry 104 (for example, to apply “special effects” or to“sharpen” the frames) to generate final output video which may be storedand/or displayed.

In one embodiment of refocusable video editing system 100, theuser/operator may be presented with a user interface including atimeline, displays or views representing conventional video effects,displays or views representing conventional video data, and/or displaysor views representing refocus video data. In this embodiment, the usermay have editing control of a plurality of features, including, forexample, the focus depth of refocusable video streams, the inclusion ofvisual effects, the timing of all elements (for example, video,refocusable video and/or effects) and/or transitions between elements.In this embodiment, the user/operator may have the option of renderingto final video. While the system is rendering portions of the finaloutput video that include refocusable video data, refocusable videoediting system 100 may render refocusable video data to intermediateoutput video data temporally prior to combining the refocusable videodata with other elements (for example, when fading from one clip toanother).

In another embodiment, with reference to FIG. 3I, refocusable videosystem 100, which includes refocusable video processing circuitry 102,may couple to video processing system 106. In some embodiments, withcontinued reference to FIG. 3I, the circuitry of video processing system106 overlaps, in whole or in part, with the circuitry of refocusablevideo processing circuitry 102. Such circuitry (for example,microprocessor circuitry) may implement conventional techniques (forexample, via conventional program(s)) that may be initialized andexecuted by refocusable video editing system 100.

In yet another embodiment, with reference to FIG. 3J, refocusable videoediting system 100 may include video processing circuitry 104 andrefocusable video processing circuitry 102. In this embodiment, videoprocessing circuitry may process refocusable video data (for example, toadjust the exposure or to de-mosaic raw light field sensor data) and,thereafter, refocusable video processing circuitry 102 may furtherprocess the refocusable video data to produce output video.

Notably, the system may implement conventional video processing and/orthe refocusable video processing in one or more stages. Such stages maynot be contiguous in that the system may implement certain conventionalvideo processing before and after certain refocusable video processing.All combinations and permutations are intended to fall within the scopeof the present inventions. For example, in one embodiment, withreference to FIG. 3K, refocusable video editing system 100 may includerefocusable video processing and two separate stages/operations usingconventional video processing. In this embodiment, refocusable videodata or frames may be processed by video processing circuitry (forexample, to apply exposure correction or to de-mosaic raw light fieldsensor data), the resulting refocusable video data may be furtherprocessed by refocusable video processing circuitry 102 to, for example,generate intermediate output video, and the intermediate output videomay be further processed by video processing circuitry 104 (for example,to apply “special effects” or to “sharpen” the frames) to generate orrender final output video.

The conventional video processing may include many effects and/orfeatures. In one embodiment, conventional video processing may includechanging color effects (for example, editing contrast/brightness, colorcast, saturation, color levels, color replacements, and/or colorinversion). In another embodiment, conventional video processing mayinclude compositing of the video data with other video and/or image data(for example, text overlays, “picture in picture”, and/or alpha blendingof other images and/or video). In another embodiment, conventional videoprocessing includes the addition of transition effects between aplurality of video segments (for example, cross-fade, fade to black,and/or transitions involving affine transforms). In another embodiment,conventional video processing includes filtering (for example, tosharpen, blur, convolve, add a “film grain” and/or reduce “noise”). Inanother embodiment, conventional video processing may include “specialeffects” (for example, CGI animation, explosions, and or “green screen”effects).

Notably, refocusable video editing system 100 may also include datastorage 22 as well as a user interface (for example, similar to or likethat illustrated in FIG. 5). The user interface may include, forexample, one or more displays, speakers, microphones, pointing devices,switches, buttons, knobs, touch screens, and/or keyboards.

With that in mind, the first subsection describes embodiments of therefocusable video editing system having one or more features, operationsand/or controls; thereafter, a more detailed description of certainembodiments including techniques that may be implemented, for example,by an operator or user. The refocusable video editing system 100 mayinclude one or more, or all of the features, operations and/or controlsor may implement one or more, or all of the techniques described herein.As such, refocusable video editing system 100 is neither limited to anysingle aspect nor embodiment, nor to any combinations and/orpermutations of such aspects and/or embodiments. Moreover, each of thefeatures, operations and/or controls of and techniques implemented byrefocusable video editing system 100 may be employed alone or incombination with one or more of the other aspects of the presentinventions and/or embodiments thereof. For the sake of brevity, many ofthose permutations and combinations will not be discussed and/orillustrated in combination herein.

Refocusable Video Editing System Layout

An exemplary refocusable video editing system 100 for refocusable videomay include, provide, implement, employ, generate and/or output (to, forexample, a display of the user interface of system 100) one or more, orall of the following components, features and/or information:

-   -   Video Timeline—The video timeline displays video input streams        that are available for a final output video.    -   Set of Key Frames—Within a refocusable video stream, key frames        of various types may be set or defined within the video data        stream (which includes a plurality of frames). The Key Frame may        include a virtual focus parameter. The virtual focus parameter        of a key frame may include information which is representative        of, determine and/or define:        -   Focus (depth or subject selection)—Select subject(s) or            focus depth.        -   Aperture—Select aperture.        -   Exposure        -   Focus Visual Effects    -   Inter-frame Interpolation—The system or user may set or define,        between key frames, the values (exposure, focus, aperture,        and/or etc.) of the video frames which are temporally disposed        between the key frames based on one or more interpolation        techniques. Notably, any interpolation technique(s) now known or        later developed may be used, including but not limited to:        nearest neighbor interpolation, linear interpolation, non-linear        interpolation, cubic interpolation, Bezier curve interpolation,        and/or spline-based interpolation.

With reference to FIG. 9, displayed are two views of the display of theuser interface of an exemplary refocusable video editing system 100which is employing refocusable video editing techniques. FIG. 9 depictstwo screen shots of the same video stream, taken at different times,displaying two different key frames.

The information displayed in the two views shown in FIG. 9 presents anexample of how the user or operator may define or set key frames andthereafter select, define and/or set certain appearance settings,including focus, exposure, and aperture. The following sections presentexemplary embodiments over how these appearance key frames may becreated and edited. Notably, the user or operator may eliminate,de-select and/or delete existing key frames.

Selection of Focus

In certain embodiments, the user or operator may select the focus basedon the depth via instructions to refocusable video editing system 100.The instructions may be communicated to the system 100, for example, viasetting the virtual focus parameter of the key frame(s) or subjecttracking. For example, the user or operator, via the user interface, mayselect, determine and/or define a virtual focus parameter in connectionwith or based on one or more subject(s), and, in response, thereto,processing circuitry in refocusable video editing system 100 may adjustthe exposure parameter of the video stream (for example, automatically)based on such subject(s) of interest (for example, change in or movementof the subject(s) of interest).

Notably, a virtual focus parameter may be any data or informationdirectly or indirectly representing a virtual focus. This data orinformation may be used, when computing an output video frame from arefocusable video data frame, to determine the virtual focus (forexample, a focus depth or a tilted plane of focus) at which the outputvideo frame is computed, generated and/or rendered. In one embodiment,the virtual focus parameter may be data or information that specifies avirtual focus depth or a virtual plane of focus. In another embodiment,the virtual focus parameter may be a spatial coordinate (for example, anX,Y position in the frame). The spatial coordinate may correlate to orrepresent the location of a subject to render in a predetermined focus(for example, in sharp focus). In another embodiment, the virtual focusparameter may be data representing a subject or object (for example, thelocation and color signature of the subject in the previous frame) thatis being tracked (for example, see Focus Tracking herein). In anotherembodiment, the virtual focus parameter may be data representing twosubjects (for example, see multiple subject/object tracking and/or TwoSubject Tracking with Tilt-Shift herein). Indeed, a virtual focusparameter may be any data or information directly or indirectlyrepresenting a virtual focus whether now known or later developed.

Virtual Focus Key Frames

The virtual focus key frames match a time in a video stream to a focusdepth in the scene. In this embodiment, the user or operator may selectthe parameter of the focus depth for one or more particular times in therefocusable video data stream. In response, refocusable video editingsystem 100 may interpolate the focus depth between key frames. Theselection of key frames and use of such frames to generate, produce,render and/or output focused video output data is shown in FIG. 10.

Notably, in FIG. 10, the user or operator may select a focus of one ormore key frames and processing circuitry, using the key frames, may beused to determine the focus depth while generating, producing, renderingand/or outputting the video data. When generating, producing, renderingand/or outputting the video data, the focus is determined using keyframes and interpolation between key frames. The interpolation betweenkey frames may be linear or non-linear. Indeed, any interpolationtechnique(s) now known or later developed may be used, including but notlimited to: nearest neighbor interpolation, linear interpolation,non-linear interpolation, cubic interpolation, Bezier curveinterpolation, and/or spline-based interpolation.

Gestures for Adding Focus Key Frames

In certain embodiments of the present inventions, the operator or usermay incorporate, insert, edit, remove, and/or add key frames through theuser interface gestures (or inputs) of system 100. In this regard, theoperator or user may choose a mode of operation of refocusable videoediting system 100, for example, through a tool selection or stateselection via the user interface, such that a gesture/input or series ofgestures/inputs incorporate(s), insert(s), edit(s), remove(s), and/oradd(s) one or more key frames according to the effect (for example,visual) that the operator is incorporating, implementing and/orproviding in the video output. For example, in response to user inputs,refocusable video editing system 100 may incorporate, insert, edit,remove, and/or add one or more key frames in the following manner:

-   -   The operator provides a gesture/input to indicate a change in        focus depth. For example, the gesture/input may be pointing and        clicking on the display of the current frame on the subject or        object that the operator would like to be brought into focus;        processing circuitry may calculate, determine and/or derive the        focus depth of that subject or object, and thereafter calculate,        adjust and/or generate the video data/frame having the new        (virtual) focus depth, which correlates to or is the depth of        the indicated subject or object. As another example, the gesture        may be clicking and dragging on a slider that represents the        currently displayed focus depth. After the gesture, the system        implements the new focus depth by calculating, adjusting,        generating, producing and/or outputting the video data/frame of        the image using the new focus position corresponding to the        subject depth where the operator has pointed. For example, let        the old focus depth be A, and the new focus depth be B.        -   “Slow focus pull”—In one exemplary embodiment of the current            inventions, the user or operator, via the user interface of            refocusable video editing system 100, defines, designates            and/or inserts a key frame at the current time with a focus            depth of B. The refocusable video editing system 100            implements a visual effect within the frames of the video            data stream that provides an extended transition in the            focus depth from the time of the previously (first) defined,            designated or set key frame to the newly (second) defined,            designated or set (that is, newly added) key frame (which            was set to a virtual focus parameter of B). The slow focus            pull is graphically illustrated in FIGS. 11A-11C. Notably,            the user or operator may implement or include one or more            “slow focus pull” techniques in a given video data stream.        -   “Jump focus pull”—In another exemplary embodiment, user or            operator defines, designates and/or inserts, via the user            interface of refocusable video editing system 100, two key            frames. The first key frame includes a focus depth parameter            of A (which may be defined, designated and/or set by the            user), at a time corresponding to a number of frames before            the second key frame. The second key frame includes a focus            depth parameter of B (again, which may be defined,            designated and/or set by the user), which may be relatively            close to the focus depth parameter of the currently            displayed frame. As such, in this embodiment, refocusable            video editing system 100 implements a visual effect that            includes a relatively rapid or quick transition (for            example, within 3-5 frames) from the previous or first focus            depth to the subsequent or second focus depth. This            embodiment is graphically illustrated in FIG. 12A-12C.            Notably, the user or operator may implement or include one            or more “jump focus pull” techniques in a given video data            stream.        -   “Peek focus pull”—In yet another exemplary embodiment, the            user may define, designate and/or insert three key frames as            well as define and/or designate the focus depth of each key            frame, via the user interface of refocusable video editing            system 100. The first and third key frames may be set to a            focus depth parameter of A, wherein there are a number of            frames before and after the second key frame. As such, in            this embodiment, refocusable video editing system 100            implements a visual effect that includes a temporary            transition from a first focus depth (A) to the second focus            depth (B) and back to the initial, first or original focus            depth (A). This embodiment is graphically illustrated in            FIG. 13A-13C. Notably, the user or operator may implement or            include one or more “peak focus pull” techniques in a given            video data stream.    -   In another exemplary embodiment, a multi-step gesture/input may        be implemented or used to, for example, provide control over the        duration of a focus pull or a portion thereof. For example, the        gesture may correspond to a click, drag and release of the        mouse. That is, the click on the current frame chooses the new        focus depth corresponding to the indicated subject. Let the old        focus depth be A, and the new focus depth be B. The drag        transitions the video backwards in time according to how far it        has been dragged. The release sets the start time of the focus        pull. Two key-frames are designated, defined inserted and/or        added: the first is at the time corresponding to the release, at        focus depth A; the second is at the time corresponding to the        click, at focus depth B. Thus, in this embodiment, the user may        define, determine and/or designate the duration of a focus pull,        or a portion thereof.

Notably, the user interface of refocusable video editing system 100 mayinclude any input device (for example, pointing device such as a mouseor trackball—see, for example, the user interface of FIG. 5). In thisregard, the pointing device is employed to allow or facilitate the useror operator to define, designate, incorporate, insert, edit, remove,and/or add one or more key frames through user-interface gestures.Indeed, where an embodiment (or example thereof) is described orexplained in the context of a mouse or trackball user interface, anyuser interface to input data or commands may be employed.

Moreover, those skilled in the art will understand through theseexamples that any number of desired focus transitions can be created,edited, and/or removed near the currently displayed depth through asingle or series of coordinated gestures/inputs. That is, the presentinventions are neither limited to any single aspect nor embodiment ofthe desired focus transitions through one or more coordinatedgestures/inputs, nor to any combinations and/or permutations of suchaspects and/or embodiments thereof. Moreover, each of the aspects of thepresent inventions, and/or embodiments thereof, may be employed alone orin combination with one or more of the other aspects of the presentinventions and/or embodiments thereof.

Focus Tracking

In one exemplary embodiment, in response to user or operator inputs,refocusable video editing system 100 may focus on one or moresubjects/objects (hereinafter collectively “subject/object” and“subjects/objects” are collectively “subject” and “subjects”respectively) and adjust and track the virtual focus parameter basedthereon. As such, in this embodiment, the user or operator designates,defines and/or identifies one or more subjects for tracking a scene andrefocusable video editing system 100 adjusts the virtual focus parameterof the designated frames to provide a focus depth such that thesubject(s) is/are always in user predetermined or defined focus (forexample, the subject(s) in the refocusable video data is/are located inthe focus plane). The video editing system 100 implements techniques totrack (from a focus perspective) the subject(s) in the video data streamand automatically select(s) the focus depth such that the subject(s)is/are in a predetermined, selected and/or defined focus (for example,always in focus) for a given or predetermined period of contiguousframes or time.

In one exemplary embodiment, the operator or user of a refocusable videoediting system may define, determine and/or select (i) the subjecttracking focus key frames by selecting a frame of interest at aparticular time, (ii) the subject(s) of interest in the frame(s), and/or(iii) the focal plane or focus for the subject or each of the subject(s)(on an individual subject or collective basis). Exemplaryimplementations of subject tracking focus embodiment processes are shownin FIGS. 14A and 14B.

These selection methods are exemplary; refocusable video editing system100 may employ other subject key frame selection process to define,determine and/or select the subject tracking focus embodiment. In oneembodiment, after the user or operator defines, determines and/orselects subject tracking key frames, refocusable video editing system100 employs the subject focus key frames to control, select, adjustand/or manage a focus depth of the refocusable video data (fromrefocusable video acquisition system 10) with which to generate,produce, render and/or output video data that includes a predeterminedfocus depth. In one exemplary embodiment, refocusable video editingsystem 100 obtains or retrieves the location of the subject of interestfrom the subject focus key frame. In this exemplary embodiment,refocusable video editing system 100 assesses, determines and/orperforms depth analysis on the location of the subject of interest inthe refocusable video data (acquired by refocusable video acquisitionsystem 10) to find the focus depth of a predetermined focus (forexample, the sharpest focus). As stated above, refocusable video editingsystem 100 modifies, changes, generates, produces, renders and/oroutputs video data (using the refocusable video data acquired byrefocusable video acquisition system 10) wherein the focus depth of suchvideo data is controlled, selected, adjusted and/or managed for apredetermined period of time (frames) to provide predeterminedsubject(s) in a predetermined focus. An exemplary embodiment of the flowof the focus subject tracking is illustrated in FIG. 15.

Notably, exemplary methods for implementing depth analysis on image dataare described in U.S. patent application Ser. No. 11/948,901 and U.S.Pat. No. 5,076,687. In one embodiment, selects and/or employs this focusdepth as the focus depth for generating, producing, rendering and/oroutputting the 2D frame.

In one embodiment, refocusable video editing system 100 may interpolatethe “correct” or predetermined depth of subject focus between subjecttracking key frames from the previous subject focus key frame bytracking the subject of interest and performing depth analysis on thetracked subject (for example, analyzing or determining the depth on thetracked subject). In one exemplary embodiment, for frames that are notsubject focus key frames, the previous subject tracking key frame in therefocusable video data may be located, and the subject of interest maybe tracked and/or identified through the intermediate frames until theframe of interest. In this exemplary embodiment, the focus depth may bedetermined by applying depth analysis on the location of the trackedsubject of interest. The flow of this embodiment is displayed in FIG.16.

Where multiple subjects are tracked, subject tracking and subject focusanalysis may be performed independently for each subject as describedabove. In an exemplary embodiment, the independent focus depths ofpredetermined focus for each subject (for example, the depth at whicheach subject is in a predetermined focus, for example, a sharpest focus)may be “combined” to determine the final focus of the frame (forexample, by selecting a virtual focal depth for the output image that isapproximately the average of the predetermined focal depths for theindividual subjects).

In one exemplary embodiment, the subject tracking system may work in thefollowing manner:

-   -   For the subject key frame, an output frame may be generated        using a small computed aperture.    -   For the subject(s) of interest chosen, regional pixel analysis        may be performed on the output frame for the region(s) of the        frame corresponding to the subject(s) of interest. In some        embodiments, the regional pixel analysis may analyze the frame        region(s) for image content, which may include the average color        of the region, shapes and colors of portions of the region(s) as        determined by an image segmentation algorithm, edge information        based on color derivatives in the region(s), and/or statistical        metrics of the contrast, brightness, and/or color variation in        the region(s).    -   For each subsequent frame, the region(s) containing the        subject(s) of interest is/are determined in the following        manner:        -   An output frame may be generated using a small computed            aperture.        -   For each of the subject(s) of interest, a set of possible            regions for the subject in the current frame may be computed            using the corresponding regions(s) of interest from the            previous frame(s). In one embodiment, the set of possible            regions in the current frame the permutations of            translations of the region(s) of interest in the previous            frame within N pixels (for example, 20 pixels), rotation            within M degrees (for example, 5 degrees), and growing or            shrinking the region within P percent (for example, 5            percent).        -   For each subject of interest and possible region for the            subject in the current frame, regional pixel analysis may be            performed on the output frame for the possible region of the            frame corresponding to the subject(s) of interest. In some            embodiments, the regional pixel analysis may analyze the            frame region(s) for image content, which may include the            average color of the region, shapes and colors of portions            of the region(s) as determined by an image segmentation            algorithm, edge information based on color derivatives in            the region(s) (for example, by creating a derivative image),            and/or statistical metrics of the contrast, brightness,            and/or color variation in the region(s).        -   For each subject of interest, the selected region of            interest for the subsequent frame may the possible region of            interest with the regional pixel analysis that most closely            matches the regional pixel analysis of the selected region            of the previous frame. In some embodiments, the most closely            matching region may be the region with the most similar            average color. In some embodiments, the most closely            matching region may be the region where the            cross-correlation score of a derivative image of the            possible region and the derivative image of the region of            interest from the previous frame is the highest.

Indeed, the subject tracking may be performed by any object trackingsystem now known or later developed, including but not limited to: ablob tracking algorithm, kernel-based tracking, contour tracking, orfeature matching. Notably, in certain specific embodiments, the varioustracking algorithms implemented by refocusable video editing system 100may be provided with output video frames with a reduced generated orcomputed aperture size.

Auto-Exposure or Predetermined Exposure Adjustment/Correction

In many lighting situations, the exposure values for one subject orscene portion may not be suitable, optimum or appropriate for anothersubject or scene portion. (See, for example, FIG. 9). With that in mind,in one exemplary embodiment, refocusable video editing system 100 maycontrol, manage, adjust and/or change the exposure parameter of apredetermined amount of refocusable video data (acquired by refocusablevideo acquisition system 10) so that, for example, the subject(s) offocus or area(s) of focus are exposed correctly and/or exposed in apredetermined manner. The exposure may be based or depend on the subjectand/or area that is/are “in focus” or designated to be in apredetermined focus within the scene. Two exemplary embodiments of theflow thereof are displayed in FIGS. 17A and 17B.

In an exemplary embodiment of auto-exposure correction, refocusablevideo editing system 100 may analyze one or more frames of refocusablevideo data (acquired by refocusable video acquisition system 10) toautomatically determine the subject or area of focus (for example, byfinding the areas of sharpest detail). Based on the automaticallydetermined subjects or areas of focus, the system may control, manage,adjust and/or change the exposure parameter of the frame (for example,to maximize image contrast on the subject of focus) and/or subsequentframes which include such subject or area. The refocusable video editingsystem 100 may employ any automatic exposure technique(s) now known orlater developed including techniques based on minimum information lossor analysis of histograms.

Notably, processing circuitry may generate, calculate and/or determine avirtual exposure parameter associated with selected refocusable videodata or frames. The processing circuitry may implement the virtualexposure parameter by generating, computing and/or outputting video dataor frames using (i) the refocusable video data temporally correspondingto the output video data or frames and (ii) the virtual exposureparameter.

Selection of Aperture

The refocusable video editing system 100 may generate output video(using refocusable video data (acquired by refocusable video acquisitionsystem 10) having a virtual aperture wherein the virtual aperture of thegenerated or computed images (i.e., output video) may be varied in, forexample, manually, automatic and/or in a predetermined manner.

Aperture Parameter Key Frames

In one exemplary embodiment, refocusable video editing system 100analyzes, determines and/or employs an aperture size (for example, whichmay be user defined or input) of key or predetermined frames. In thisembodiment, the user or operator may determine key frames and set,define and/or determine aperture size and/or aperture shape parameter(s)associated with such key or predetermined frames. The refocusable videoediting system 100 may modify and/or change the aperture size and/orshape parameters of the video data associated with one or more frames ofrefocusable video data (acquired by refocusable video acquisition system10) in accordance with such parameter(s). That is, processing circuitrymay generate, calculate and/or determine a virtual aperture size and/orvirtual aperture shape parameter(s) associated with such key frames forthe refocusable video data associated with the key frames. Thereafter,processing circuitry may implement the virtual aperture size and/orvirtual aperture shape parameter(s) in a manner discussed herein inconnection with key frames. That is, processing circuitry may generate,compute and/or output video data or frames using (i) the refocusablevideo data temporally corresponding to the output video data or framesand (ii) corresponding virtual aperture size and/or virtual apertureshape.

Moreover, refocusable video editing system 100, based on the aperturesize and/or aperture shape parameter(s) of the key frames, mayinterpolate between the key frames to determine the aperture size and/oraperture shape parameters for all times in the video. Briefly, withreference to FIG. 18, a flow of a selection of aperture key frames,using the key frames while generating, outputting and/or rendering the2D video as output video data. Aperture key frames may be selected bythe user or operator. When rendering, refocusable video editing system100 may determine the aperture to use based on the parameters of the keyframes and interpolation between key frames. The processing circuitrymay implement the virtual aperture size and inter-frame interpolation ina manner discussed herein in connection with key frames. In this regard,processing circuitry may generate, compute and/or output video data orframes using (i) the refocusable video data temporally corresponding tothe output video data or frames and (ii) the corresponding virtualaperture size and (iii) inter-frame interpolation, which may modify,modulate and/or change the virtual aperture size for a given outputvideo data or frames.

Notably, any interpolation technique now known or later developed may beused, including but not limited to: nearest neighbor interpolation,linear interpolation, non-linear interpolation, cubic interpolation,Bezier curve interpolation, and spline-based interpolation.

Aperture Selection to Include Subjects of Interest

In yet another exemplary embodiment of the present inventions,refocusable video editing system 100 set, employ and/or adjust the(virtual) aperture of the refocusable video data to include one or moresubjects of interest contained or imaged in such refocusable video data.The refocusable video editing system 100 may automatically select anaperture (and, in another embodiment, a focal depth) that maintains andprovides the subject(s) of interest in a given, selected and/orpredetermined focus by modifying the refocusable video data (acquired bythe refocusable acquisition system 10) such that that all subjects areincluded within the resulting depth of field). In this regard,refocusable video editing system 100 may change or varying the virtualaperture size and virtual focal depth of the refocusable video data toprovide the subjects to be included within the resulting depth of field.That is, processing circuitry may generate, calculate and/or determine avirtual aperture size and virtual focal depth for the refocusable videodata that provides the subjects to be included within the resultingdepth of field. The processing circuitry may generate, compute and/oroutput video data or frames using the refocusable video data and suchvirtual aperture size and virtual focal depth.

In addition thereto, in one embodiment, the user or operator mayinitially establish or select an aperture and the system may,thereafter, maintain the subject(s) of interest in a given, selected orpredetermined focus. In some embodiments, the user or operator mayestablish or select a minimum focus score, as determined by a focusanalysis metric, for each of the subject(s) of interest. In theseembodiments, the system may choose the maximum aperture (and hence, themost limited depth of field) that maintains the minimum focus score forthe subject(s) of interest. In these embodiments, scores for any focusmetric now known or later developed may be used.

In one embodiment, the user or operator may identify or indicatesubject(s) of interest, via the user interface of refocusable videoediting system 100, by outlining such subject(s) as described above (forexample, two or more regions corresponding to subjects of interest). Inanother exemplary embodiment, refocusable video editing system 100 mayautomatically determine subject(s) of interest using analysis techniquesas described herein.

Visual Effects

In another set of exemplary embodiments, refocusable video editingsystem 100, using the refocusable video data and in response to userinputs, implements and/or renders visual effects into output video.Here, the input data is refocusable video data, and the output is outputvideo that contains such visual effects. Exemplary embodiments of thevisual effects are provided immediately below, however, it should benoted that such exemplary embodiments are not intended to limit thescope of the present invention.

Two Subject Tracking with Tilt-Shift

In an exemplary embodiment, refocusable video editing system 100 maygenerate output video data having at least one focal plane that isskewed (or off-axis) relative to a second focal plane (for example, thefocal plane of the optics of the refocusable video acquisition unit 12during acquisition, sampling and/or capture of the refocusable videodata. For example, with reference to FIG. 19, in one embodiment, theuser or operator may select two or more subjects of focus wherein afirst subject is in a first focal plane which is parallel to the focalplane of refocusable video acquisition unit 12 and a second subject isin a second focal plane which is different from the first focal planeand also parallel to the focal plane of refocusable video acquisitionunit 12. Here, processing circuitry calculates, determines and/orderives virtual focus characteristic(s) or parameter(s) including thelocation of the focal plane(s), aperture(s) and/or focus effect(s) (thatis, where to focus and/or the amount of focus) to provide a focal planethat is skewed (or off-axis) relative to a second focal plane (forexample, the focal plane of the optics of the refocusable videoacquisition unit during acquisition, sampling and/or capture of therefocusable video data). The refocusable video editing system 100, forexample, may simulate a tilt-shift lens in computing output video usingthe refocusable video data. In this way, refocusable video editingsystem 100 may generate output video having the subjects in apredetermined focus (for example, both subject in sharp focus),regardless of their differences in focal depth. In the case of threesubjects, the system and technique may choose a desired focal plane that“passes through” the three subjects.

Notably, in one exemplary embodiment, the user or operator may createand configure such “tilt-shift focus” key frames in the video, andthereafter system and technique may interpolate between the various keyframes (for example, by automatically tracking subjects between keyframes to maintain selected subjects in a predetermined focus, forexample, a sharpest focus).

In one exemplary embodiment, refocusable video editing system 100 maydecompose the plane equation for key frames into a composition of arotation and a translation; the refocusable video editing system 100 mayinterpolate the rotations in quaternion space, and interpolates thetranslation in Euclidean space. The refocusable video editing system100, and technique implemented thereby, may employ any interpolationtechnique now known or later developed, including but not limited to thefollowing: nearest neighbor interpolation, linear interpolation,non-linear interpolation, cubic interpolation, Bezier curveinterpolation, and spline-based interpolation.

Subject Isolation

In another exemplary embodiment, refocusable video editing system 100,in response to user or operator inputs, may visually isolate one or moresubjects to, for example, to direct a viewer's gaze onto the subjectregion. In this embodiment, the user or operator may specify one or moreregions of the image where one or more subjects are to be visuallyisolated, using a focus effect. The processing circuitry in system 100may thereafter generate, compute and/or output video data wherein thepredetermined or identified subject(s) are in a first focus (forexample, in focus), and the regions surrounding the subject(s) are in asecond focus (for example, blurred). This embodiment may simulate anoptical aberration in connection with the refocusable video acquisitionsystem 10.

In one embodiment, the user or operator may define and/or set key framesthat identify these regions, and processing circuitry in refocusablevideo editing system 100 may generate, compute and/or output video databy interpolating between such key frames. In another exemplaryembodiment, the key frames comprise a 3-tuple (x,y,r), where (x,y) isthe center of the region, and r is a radius of the desired focus region.The processing circuitry in refocusable video editing system 100, andtechnique implemented thereby may interpolate between the 3-tuples ateach key frame to determine the values at any point in the video.Notably, any interpolation technique now known or later developed may beused, including but not limited to the following: nearest neighborinterpolation, linear interpolation, non-linear interpolation, cubicinterpolation, Bezier curve interpolation, and spline-basedinterpolation.

In another exemplary embodiment, refocusable video editing system 100may isolate the subject by generating and/or computing video data thatwould have been produced by a lens (of the optics of refocusable videoacquisition system 10) with an aberration, wherein such lens isoptically focused on the subject region, and other areas are blurred tooptical lens aberration. In this regard, processing circuitry generates,calculates and/or determines a virtual focus for the refocusable videodata that provides the predetermined aberration. The processingcircuitry may then generate, compute and/or output video data or framesusing the refocusable video data and such virtual focus.

In yet another exemplary embodiment, the aberration corresponds to theaberration from a lens where the optical axis is tilted to approximatelypoint at the subject of interest, and the optical focus approximatelyintersects the subject of interest. In a specific exemplary embodiment,the lens may be a single lens element, such as a planoconvex or doubleconvex lens. In one exemplary embodiment, the lens may be a doubletlens. In this embodiment, processing circuitry generates, calculatesand/or determines a virtual focus for the refocusable video data thatprovides the predetermined aberration as described immediately above.Thereafter, processing circuitry may generate, compute and/or outputvideo data or frames using the refocusable video data and such virtualfocus.

Notably, those skilled in the art will understand through these examplesthat any number of desired lenses or systems of lenses may be modeled.That is, the present inventions are neither limited to any single set oflenses or system of lenses nor embodiment of simulated lens aberrations.Moreover, each of the aspects of the present inventions, and/orembodiments thereof, may be employed alone or in combination with one ormore of the other aspects of the present inventions and/or embodimentsthereof.

In yet another exemplary embodiment, the user or operator may select,define, identify or indicate, via the user interface of refocusablevideo editing system 100, a subject to be tracked, for example, asdescribed herein. The system and technique may thereafter compute outputvideo data having frames where the subject is maintained or kept infocus (or in a certain predetermined or first focus), and surroundingregions are maintained or kept is a second or different focus (forexample, blurred out), for example with the optical aberration effect asdescribed above.

Split-Screen Focus

In another exemplary embodiment, refocusable video editing system 100,in response to user or operator inputs which define a plurality of keyframes wherein each key frame includes a plurality of N regions andfocus depth of each region may be controlled or defined separately. Therefocusable video editing system 100, and technique implemented therein,may interpolate the focus in each of the N regions independently insuccessive or preceding frames of the refocusable video data. As before,any interpolation technique now known or later developed may be used,including but not limited to: nearest neighbor interpolation, linearinterpolation, non-linear interpolation, cubic interpolation, Beziercurve interpolation, and spline-based interpolation.

Notably, any method or system now known or later developed fordetermining focus and/or generating, producing or creating output videodata may be performed in any of the N regions. Indeed, each region mayuse the same, similar, or different systems for determining and/orgenerating, producing or creating output video data.

In one exemplary embodiment, N equals 2, the separation between the tworegions is a line, and the focus is set independently on each portion ofthe screen in order to produce a “spit-screen” focus effect (See FIG.26). In one specific exemplary embodiment, N equals 2, and each regionmay use “Focus Subject Tracking”, presented herein, to maintain focus ona separate object in both of the regions.

Slow-Motion Focus Pull

In yet another exemplary embodiment, refocusable video editing system100, in response to user or operator inputs, may generate, produceand/or output video data, using the refocusable video data (acquired byrefocusable video data acquisition unit 12) that includes a real-timefocus pull, including slow-motion video. In this embodiment, the user oroperator, via the user interface of refocusable video editing system100, may designate, define and/or indicate a portion or section of thevideo data to be rendered in slow motion, for example, by indicating theduration of input footage and the duration of output rendered playbackvideo (for example, where 1 second of footage takes 20 seconds ofplayback). In addition, the user or operator may designate, defineand/or indicate for this speed-modified video sequence a desired changein appearance of focus, aperture, exposure, and/or etc. of therefocusable video data, for example, using one or more of the techniquesdescribed herein. (See, for example, the technique of “Focus SubjectTracking” or “Two Subject Tracking with Tilt-Shift”). In response,refocusable video editing system 100 generates, produces, renders and/oroutputs the video data at the desired rates with the desired change inappearance.

In some embodiments, the refocusable video data may have been originallycaptured at a higher frame rate (for example, 120 frames per second)than the frame rate desired for output video (for example, 30 frames persecond). In these embodiments, the output video speed may appear in“slow motion” by rendering the originally captured refocusable framesover a longer duration than the time of capture (for example, 120refocusable frames captured in 1 second may be rendered as 120 outputframes lasting for 4 seconds), using one or more of the techniquesdescribed herein. (See, for example, the technique of “Focus SubjectTracking” or “Two Subject Tracking with Tilt-Shift”). In otherembodiments, the refocusable video data may have been originallycaptured at the same or similar frame rate desired for output video. Inthese embodiments, it may be desirable during “slow motion” to maintaina consistent frame rate for output video (for example, 30 frames persecond) by interpolating between frames in the refocusable video data.In one specific embodiment, frames in the refocusable video data may berepeated to maintain the proper output frame rate. In another specificembodiment, the refocusable video data may be rendered at the originallycaptured frame rate, and intermediate frames may be generated byinterpolating between output video frames (for example, by blending orby applying motion and/or scene analysis to create intermediate frames).Indeed, any video frame interpolation method(s) now known or laterdeveloped may be used.

Notably, Split Screen focus and Slow-motion Focus Pull embodimentsdescribed above, processing circuitry generates, calculates and/ordetermines virtual focus parameters (including a virtual focus) for therefocusable video data that provides the predetermined effect asdescribed immediately above. Thereafter, processing circuitry maygenerate, compute and/or output video data or frames using therefocusable video data and such virtual focus parameter (includingvirtual focus).

Painted Focus

In another exemplary embodiment, refocusable video editing system 100employs a technique that allows the user or operator to “paint” one ormore different depths of focus for one or more areas of a frame tocreate painted focus key frames. For example, the user or operator may,via the user interface of refocusable video editing system 100, use a“brush”-like tool that sets, defines and/or establishes a focus depth ofone or more areas of a given frame “directly” on the rendition of theframe presented to the user on a display of the user interface. Inresponse, refocusable video editing system 100 defines, establishes anddesignates a focus depth for each of the one or more areas of the frame.In this way, the user or operator, via the user interface of refocusablevideo editing system 100, “paints” or defines the depth(s) of focus ofsuch one or more areas “directly” onto a representation of the frame.The system may interpolate focus for each pixel independently betweenkey frames using, for example, nearest neighbor interpolation, linearinterpolation, non-linear interpolation, cubic interpolation, Beziercurve interpolation, and spline-based interpolation. Notably, anyinterpolation technique now known or later developed may be used. (See,for example, U.S. Patent Application Publication 2008/0131019 (U.S.patent application Ser. No. 11/948,901).

Integrated Refocusable Video System

As noted above, refocusable video editing systems 100 and techniquesimplemented thereby, may be integrated into refocusable videoacquisition system 10, and, as such, some, most or all of the processingmay be performed on/in acquisition system 10 or unit 12 and thatprocessing not performed on/in acquisition system 10 or unit 12 may beperformed in a physically separate and external system/device. Oneexemplary embodiment of this system is shown in FIGS. 3B, 3E and 3G and20.

Recording

Also noted above, refocusable video acquisition unit may operate as astill camera and a video camera according to one or more, or all of theembodiments described herein. Indeed, in certain embodiments, therefocusable video acquisition unit may be configured as a conventionalvideo acquisition unit (see, for example, FIGS. 2D, 2F and 4A-4E).

Selectable Refocusing Power System (On/Off System)

In this exemplary embodiment, the refocusable video acquisition unit 12has a method for shooting in conventional 2D imaging mode, or inrefocusable mode. A method for implementing such a selectable refocusingpower system utilizing a light field recording system is described inPatent Cooperation Treaty Application WO 2007/092545 A2.

As discussed above, in one exemplary embodiment, refocusable videoacquisition unit 12 is switched between modes as follows: when shootingin still mode, refocusable video acquisition unit 12 is configured toshoot in conventional 2D imaging mode; when shooting in video mode,refocusable video acquisition unit 12 functions or operates as arefocusable video camera that records light field video. (See, forexample, FIG. 2C and FIG. 2D).

Focus Planned Recording

In another exemplary embodiment, the system and technique of the presentinventions may include “Focus Planning” to determine the correct,desired and/or predetermined location of focus. In this regard, in oneembodiment, processing circuitry of refocusable video acquisition system10 and/or refocusable video editing systems 100 may evaluate or analyzea plurality of frames of the refocusable video data, including (i) datawhich is representative of video frames that precede the current frameand (ii) data which is representative of video frame that follow thecurrent frame. Notably, in the context of “Focus Planned Recording”, the“current frame” refers to a refocusable frame that is being processed bythe refocusable video acquisition system 10 in order to render a 2Doutput video frame. In this sense, the “current frame” may be a framecaptured at a point in the past relative to events that are occurring“live”. In one embodiment, refocusable video acquisition system 10and/or refocusable video editing systems 100 includes memory (forexample, one or more buffers) to store light field data or informationof the refocusable input video data or stream corresponding to (i) videoframes that precede the current frame and (ii) video frames that followthe current frame. Using such refocusable video data, data processingcircuitry may perform a temporal video analysis thereof to calculate,determine and/or obtain the predetermined, correct or desired focusdepth. An exemplary embodiment of flow of the focus planning is shown inFIG. 21A.

With reference to FIG. 21A, in one embodiment of refocusable dataacquisition or capture using Focus Planning according to the presentinventions, refocusable video acquisition system 10 and/or refocusablevideo editing systems 100 stores/buffers refocusable video datacorresponding to a plurality of video frames (for example, temporallycontiguous refocusable light field video data frames) including, forexample, video frames before and after the video frame to render,acquire and/or output. Such systems 10 and/or 100 use(s) the refocusablevideo data of the preceding and subsequent frames to calculate,determine and/or obtain virtual focus characteristic(s) or parameter(s)including the location of the focal plane(s), aperture(s) and/or focuseffect(s) (that is, where to focus and/or the amount of focus).

In one embodiment, during operation, N seconds of refocusable videolight field data (corresponding to a plurality of refocusable videoframes) are stored (for example, buffered) for video frames that occurafter a given video frame to render, acquire and/or output, and Mseconds of refocusable video data (corresponding to refocusable videoframes) that occur before the given video frame to render, acquireand/or output. For example, systems 10 and/or 100 may store (i) 5seconds of refocusable video data corresponding to frames that occurredbefore the given video frame to render, acquire and/or output and (ii) 5seconds of refocusable video data corresponding to frames that occurredafter the given video frame to render, acquire and/or output.

Notably, N and M may be equal. Moreover, data which is representative ofthe amount of time may be user or operator programmable (via the userinterface wherein such user input is stored in memory, for example, aregister or buffer) and/or system defined (for example, based on theamount of memory available to implement Focus Planning).

Notably, in some embodiments, refocusable video acquisition system 10may reach a state where the entire N+M second buffer has filled withrefocusable video data while the system continues to acquire additionalrefocusable video data. During this “steady state” operation, one ormore refocusable video frames may be deleted from the buffer or beoverwritten by newly acquired refocusable video frames. In someembodiments, the oldest refocusable video frames in the buffer aredeleted and/or overwritten. In a specific embodiment, in “steady state”the system may continually remove and/or overwrite the single “oldest”or earliest acquired refocusable video frame and acquire a newrefocusable video frame (whether concurrently with or after the removalof the refocusable video frame from the memory).

In another embodiment, refocusable video acquisition system 10 and/orrefocusable video editing systems 100 employ(s) the refocusable videodata of the frames acquired after a selected video frame to derive,calculate and/or determine virtual focus characteristic(s) orparameter(s) (for example, data which is representative of the locationof the focal plane(s), focus depth(s), size and shape of the aperture(s)and/or focus effect(s) (that is, where to focus and/or the amount offocus)) of one or more previously acquired video data frames. In thisregard, processing circuitry in systems 10 and/or 100, using or based ondata corresponding to video frames acquired after refocusable video datacorresponding to the selected video frame, may derive, calculate and/ordetermine virtual focus characteristic(s) or parameter(s) associatedwith the previously acquired refocusable video data. In response,refocusable video acquisition system 10 and/or refocusable video editingsystems 100 generates and outputs (for example, to internal or externalmemory) video frame(s) or data using (i) the virtual focuscharacteristic(s) or parameter(s) (for example, data which isrepresentative of the characteristic or parameter) and (ii) therefocusable video data corresponding to or associated of thecorresponding video frame.

For example, in one embodiment, the focus depth of one or more selectedvideo refocusable video data frames may be changed, determined, definedand/or re-defined after acquisition of the corresponding refocusablelight field video data based on or using refocusable video datacorresponding to or associated with the video frames acquired after theselected video refocusable video data frames. That is, in thisembodiment, the focus depth of the virtual focus characteristic(s) orparameter(s) of a selected video frame may be determined based on orusing data corresponding to video frames acquired after refocusablevideo data corresponding to the selected video frame.

Moreover, until redefined, the virtual focus characteristic or parametermay also define or determine the focus depth(s) of those video framesobtained, captured and/or acquired after the video frame of the one ormore selected video frames. That is, if and until the virtual focuscharacteristic or parameter are redefined, the output video frames ordata may be determined, generated and/or calculated using (i) the“current” virtual focus characteristic or parameter and (ii) therefocusable video data corresponding to or associated of the videoframes to be rendered or output.

Notably, processing circuitry in refocusable video acquisition system 10and/or refocusable video editing systems 100 may calculate, determineand/or generate the virtual focus characteristic(s) or parameter(s)which may include data which is representative of or defines thelocation of the focal plane(s), focus depth(s), size and shape of theaperture(s) and/or focus effect(s) (that is, where to focus and/or theamount of focus)) of one or more refocusable video frames or data. Inthis regard, the processing circuitry may calculate, determine and/orgenerate a virtual focus characteristic or parameter in accordance with,for example, subject/object focus tracking (for example, as describedherein), maintaining a predetermined focus depth of an area of theacquired scene (for example, as described herein), the manner, form ortype of focus transition (similar to that as described herein inconnection with key frames), and/or in response to one or more userinputs (which may be, for example, indicative of subject/object trackingor a position or location in the scene, or an absolute focus depth).

For example, in one embodiment, in connection with subject/object focustracking (which may be system or user defined), the processing circuitrycalculates, determines and/or generates a virtual focus depth of avirtual focus characteristic or parameter of the video frames based on afocus depth of one or more subjects/objects in a scene. In this regard,the virtual focus characteristic or parameter of the video frameprovides a predetermined virtual focus depth relative to one or moresubjects/objects for a plurality of video frames such that the virtualfocus characteristic or parameter provides or maintains the one or moresubjects/objects in a predetermined focus (whether relative or absolutefocus) over the plurality of frames. Again, the system or user mayimplement subject/object tracking and/or identify the subject/object inconnection with the subject/object tracking. Moreover, thesubject/object to be tracked may be initially identified in or usingrefocusable light field video data which is acquired after acquisitionof the refocusable light field video data and the system correspondingto the video data to be output or rendered. In addition, the systemthereafter determines a virtual focus depth of the virtual focuscharacteristic or parameter, on a frame by frame basis, to provide apredetermined virtual focus depth relative to one or moresubjects/objects such that the virtual focus characteristic or parameterprovides or maintains the one or more subjects/objects in apredetermined focus (whether relative or absolute focus) over theplurality of frames—notwithstanding the fact that the correspondingrefocusable light field video data or frames were acquired using anoptical focus that is unrelated to and/or different from the virtualfocus depth of the individual frames which provide a predeterminedvirtual focus depth relative to one or more subjects/objects.

In an exemplary embodiment, with reference to FIG. 21B, in operation,refocusable video acquisition unit 12 acquires, samples and/or capturesof refocusable light field video data (which correspond to a pluralityof temporally contiguous refocusable video data frames) and data whichis representative thereof (for example, the refocusable light fieldvideo data itself or a compressed version thereof) is stored in a memorybuffer (see, (F)). Each frame of the refocusable light field video dataincludes an optical focus that corresponds to or is associated with anoptical focus of refocusable video acquisition unit 12 duringacquisition of the refocusable light field video data. The memory buffermay store up to a predetermined amount of temporally contiguous framesof refocusable video data which corresponds to, for example, three ormore seconds and/or less than ten seconds of refocusable light fieldvideo data or frames.

Thereafter, refocusable video acquisition system 10 and/or refocusablevideo editing system 100 generate(s) the output video data or framescorresponding to the refocusable video data frames and a virtual focuscharacteristic or parameter. For example, refocusable video acquisitionsystem 10 and/or refocusable video editing systems 100 may generateoutput video data or frames having a virtual focus depth that isdifferent from and/or unrelated to the optical focus depth of thecorresponding refocusable light field video data or frames. In thisembodiment, the virtual focus depth of certain video data frames (i.e.,those frames which are acquired before determination and/orimplementation of the virtual focus characteristic or parameter) aredetermined by the processing circuitry using refocusable light fieldvideo data which is acquired after the refocusable light field videodata or frames corresponding to the video frames or data to be renderedor output. Notably, the system may remove and/or overwrite therefocusable video frames/data corresponding to the rendered videoframes/data and store/buffer newly acquired refocusable video frames(whether concurrently with or after removal of the refocusable videoframes/data corresponding to the rendered video frames/data).

In one embodiment, systems 10 and/or 100 receive, calculate, determineand/or generate the virtual focus characteristic or parameter forrefocusable video data or frames stored in the memory buffer afteracquisition and storage in the memory buffer. For example, in oneembodiment, processing circuitry in systems 10 and/or 100 calculates,determines and/or generates, for one or more previously acquiredrefocusable video frames or data, the virtual focus characteristic orparameter which determines, for example, the virtual focus depth(s) ofthe one or more refocusable video frames or data. Thereafter, systems 10and/or 100 generate the output video data or frames corresponding to therefocusable video data frames and the virtual focus characteristic orparameter (for example, the virtual focus depth(s)).

In one exemplary embodiment, upon implementing subject/object focustracking, the processing circuitry of systems 10 and/or 100 calculates,determines and/or generates a virtual focus depth of the virtual focuscharacteristic or parameter for a plurality of refocusable light fieldvideo data or frames which provide or maintain the one or moresubjects/objects in a predetermined focus in the output video data orframe. The processing circuitry analyzes the refocusable light fieldvideo data or frames to identify the one or more subjects/objects insuch refocusable light field video data or frames and, in responsethereto, generates a virtual focus depth which provides or maintains theone or more subjects/objects in a predetermined focus (relative to theone or more subjects) in the output video data or frame. In oneembodiment, the processing circuitry generates output data or framesusing (i) the virtual focus depth of the virtual focus characteristic orparameter and (ii) the associated or corresponding refocusable lightfield video data or frames in order to render, output or provide outputvideo data or frames having a predetermined focus relative to the one ormore subjects/objects. Accordingly, as the one or more subjects/objectsmove within the scene, the virtual focus depth of the virtual focuscharacteristic or parameter may change to provide or maintain the one ormore subjects/objects in a predetermined focus over the plurality offrames.

Notably, in one embodiment, when the virtual focus characteristic orparameter is determined, refocusable video acquisition system 10 and/orrefocusable video editing systems 100 generates or renders output videodata or frames using such virtual focus characteristic or parameter andthe refocusable video data frames then present in the memory buffer. Forexample, in those embodiments where the virtual focus characteristic orparameter is indicative of maintaining a predetermined focus depth of anarea of the scene (for example, an “x,y” location in the scenereflective of a user input), the processing circuitry may determine avirtual focus depth associated with the area of the scene (for example,a virtual focus depth which provides a predetermined focus (for example,a virtual focus depth that provides such area “in-focus”) in connectionwith that area of the scene). Thereafter, systems 10 and/or 100 maygenerate output video data or frames using the virtual focus depthassociated with the area of the scene and the refocusable video data orframes stored or contained in the memory buffer (which correspond tovideo data or frames acquired before the refocusable video data orframes used to determine a virtual focus depth associated with the areaof the scene). Under these circumstances, the output video data orframes is generated using a virtual focus depth that corresponds to thefocus depth associated with the area of the scene which was defined ordetermined after acquisition of the refocusable video data or framesstored or contained in the memory buffer. In addition, the output videodata or frames includes a virtual focus depth—notwithstanding the factthat the corresponding refocusable video data or frames were acquiredusing an optical focus that is unrelated to and/or different from thevirtual focus depth.

In another exemplary embodiment, where the virtual focus characteristicor parameter is indicative or representative of subject/object focustracking, the processing circuitry analyzes the previously acquiredrefocusable video data or frames (which, in this example, are stored inthe memory buffer) to identify the subject/object therein andcalculates, determines and/or generates an appropriated, selected and/orsuitable virtual focus depth of the virtual focus characteristic orparameter to provide or maintain the subject/object in the predeterminedfocus. The refocusable video acquisition system 10 and/or refocusablevideo editing systems 100 employs the virtual focus depth for theassociated refocusable video data or frame to generate an output videodata or frame which provides or maintains the subject/object in apredetermined focus (whether relative or absolute focus) over theplurality of frames. The systems 10 and/or 100 generate the output videodata or frames having the subject/object in the predetermined focususing (i) a virtual focus depth which provides the subject/object in thepredetermined focus and (ii) the refocusable video data orframes—notwithstanding the fact that the refocusable video data orframes were acquired using an optical focus that is unrelated to and/ordifferent from the virtual focus depth which provides the subject/objectin the predetermined focus. Notably, in this embodiment, as thesubject/object moves within the scene, the virtual focus depth of thevirtual focus characteristic or parameter may change in order toprovide, generate, output and/or render video data or frames having apredetermined virtual focus depth relative to the one or more subjects(which thereby maintains the subject/object in the predetermined focusover the plurality of frames).

As noted above, the processing circuitry may calculate, determine and/orgenerate a virtual focus characteristic or parameter in accordance with,for example, the manner, form or type of virtual focus transition(similar to that as described herein in connection with key frames). Inthis regard, the manner, form or type of focus transition may define ordetermine a virtual focus transition(s) (for example, the Slow FocusPull, Jump Focus Pull, Peek Focus Pull, described herein (see, FIGS.11A-13C)) from a particular virtual focus depth of a first virtual focusparameter to a second virtual focus depth of a second virtual focusparameter.

For example, with reference to FIG. 21C, in an embodiment including arefocusable video acquisition system 10 incorporating Focus Planning(described herein), the processing circuitry may implement transitionsfrom virtual focus parameter 1 to virtual focus parameter 2 inaccordance with a virtual focus transition. The characteristics of thevirtual focus transition include a time period (or a number of frames)between the transition and the focus manner, form or type (for example,the Slow Focus Pull, Jump Focus Pull, Peek Focus Pull). The transitionmay be implemented concurrently with or after (for example, immediatelyafter) the determination or implementation of the “new” virtual focusparameter (virtual focus parameter 2 in the illustration of FIG. 21C).Indeed, one or more of these variables may be controlled, defined and/ordetermined by, for example, the user via a user input (wherein, forexample, such data/information may be stored in a register) and/orsystem controlled, defined and/or determined.

Another exemplary embodiment is illustrated in FIG. 21D. With referenceto FIG. 21D, in one exemplary embodiment of refocusable data acquisitionor capture using Focus Planning according to certain aspects of thepresent inventions, refocusable video acquisition system 10 and/orrefocusable video editing systems 100 may perform frame event analysisand output rendering operations. At or near the time refocusable lightfield video frames are captured, the frames may be analyzed for eventsof interest. For example, the system(s) may analyze and record any typeof events, including but not limited to:

-   -   Face detection, tracking, analysis and/or recognition may be        performed to determine if any individuals and/or specific        individuals are entering the field of view, are present,        performing a specific action (for example, speaking or        gesturing) and/or leaving the field of view.    -   Object detection, tracking, analysis and/or recognition may be        performed to determine if any objects and/or specific objects        are entering the field of view, are present, performing a        specific action (for example, moving quickly) and/or leaving the        field of view.    -   Scene analysis and/or recognition may be performed to determine        the current scene type (for example, outdoors) and/or any        changes to the scene.        Indeed, any type of video, image and/or light field analysis        presented herein, whether now known or later developed, may be        performed to generate events of interest. These events may be        stored in memory for P seconds prior to the current frame and Q        seconds following the current frame. In some embodiments, P is        equal to the total time before the current frame that the system        has been recording (for example, all events since prior to the        current frame since the user/operator last started recording).        In some embodiments, Q may be equal to M. In some embodiments, N        may be 0, indicating that only events and not refocusable frames        are stored prior to the current frame.

With reference to FIG. 21E, in one exemplary embodiment, a buffer may belocated in a first memory on refocusable video acquisition unit 12 andmay contain K refocusable frames (where, for example, K may equal Ntimes the acquisition frame rate) after the current frame and all eventsof interest for the current recording session. In this embodiment,refocusable frames may be rendered to output video using, in part, thecurrent refocusable video frame or data and one, some or all events ofinterest stored in the first memory. These output video frames may berendered to a second memory. Notably, the second memory may be the sameas, or a portion of the first memory (for example, internal memory) ormay be different. After rendering, generating and/or outputting theoutput video frame or data, the refocusable frame or data correspondingto such output video frame or data may be deleted, removed, and/oroverwritten from/in the memory buffer. Notably, the output video frameor data may be stored in a second memory (for example, an externalmemory having a larger capacity), for example, non-volatile storage (forexample, Flash memory or a hard drive).

When rendering the current frame to output video, this set of events maybe used and/or analyzed to calculate, determine and/or obtain a virtualfocus characteristic or parameter (for example, the location of thefocal plane(s), aperture(s) and/or focus effect(s) (that is, where tofocus and/or the amount of focus)). In one specific embodiment, thesystem may adjust the focus (define a virtual focus plane) on person(s)as they enter the scene or a field of view of the video (hereinaftercollectively, the scene). In this case, it may be desirable to begin toshift the plane of focus gradually over X seconds (for example, over 2seconds) so that the person(s) are in focus as they appear. Indeed, anyset of rules, system and/or method(s) of analysis, whether now known orlater developed, may be used to calculate, determine and/or obtain thelocation of the focal plane(s), aperture(s) and/or focus effect(s).

In some embodiments, the system may include, determine and/or maintaindata representing a timeline of virtual focus depth values, which at anytime may include data representing the virtual focus depth of allrefocusable frames currently in the memory buffer. At the time thecurrent refocusable frame is rendered to output video, the virtual focusparameter may be the timeline, and the current refocusable frame may berendered with the focus depth of time corresponding to the currentrefocusable frame to render according to the data representing a focustimeline. As new frames are acquired and events of interest aregenerated, the data representing the timeline of focus depth values maybe adjusted and/or modified based on the events of interest. In oneembodiment, the system may select key frames of focus based on selectedevents of interest (for example, a subject begins to speak) and mayinsert focus transition(s) (for example, the Slow Focus Pull, describedherein) temporally prior to or after the event of interest so that therendered video contains focus transitions and renders the framescorresponding to the selected events of interest in predetermined focus(for example, sharp focus).

With reference to FIGS. 27A and 27B, these figures illustrate exemplaryfocus timelines at two different and partially overlapping times duringacquisition. Notably, the time periods covered in FIGS. 27A and 27B areillustrated as overlapping, and the time covered in FIG. 27B that is notcovered in FIG. 27A occurs temporally after the time period covered inFIG. 27A. In FIG. 27A, the focus timeline is set to a first focus depth(for example, the focus depth of the background (for example, a wall ora mountain)) for all times represented in the timeline. Also displayedin FIG. 27A is a time of subject entry (TE), which occurs (for example,when a subject enters the scene) temporally just after the time ofacquisition. Notably, with respect to FIG. 27A, TE has not yet occurred.Temporally between the acquisition times in FIGS. 27A and 27B (AT-A andAT-B), TE occurs (for example, a subject of interest enters the scene).As a result of the subject entry, two key frames may be inserted intothe focus timeline, one at the time of transition start and one at TE,that may allow for a transition of focus from the first depth to asecond focus depth (for example, the focus depth that brings the subjectinto sharp focus). The resulting focus timeline is displayed in FIG.27B, wherein the focus timeline is conceptually composed of threepartitions: the earliest period of time has a focus depth correspondingto the background, the latest part, beginning at TE, has a focus depthcorresponding to a subject that has entered the scene, and the middlepart has a transition in focus from the background to the subject ofinterest. Notably, the focus depth of the focus timeline at AcquisitionTime A (AT-A) changes between the times represented in FIGS. 27A and27B, due to an event that occurred temporally after AT-A. As a result,the output video frame temporally corresponding with AT-A may berendered using a focus depth determined by an event that occurs in thefuture relative to AT-A, in a manner that may appear as if the systemhad knowledge future events.

Using that refocusable video data, the data processing circuitry maygenerate output video and present the user or operator (via the userinterface) with video frames corresponding to before and after the frameto render, acquire and/or output. Based on such video, the user oroperator may define, set and/or determine the focus characteristics ofacquisition or capture by refocusable video acquisition unit 12. Inresponse thereto, refocusable video acquisition unit 12 implements theselected focus and acquires, samples, stores and/or records refocusablevideo data based on the selected focus. Indeed, in one embodiment, inaddition to implementing the selected focus of the virtual focuscharacteristic or parameter in connection with the video frame or datato be rendered, acquired and/or output, systems 10 and/or 100 implementthe selected focus in connection with the preceding and/or subsequentframes.

The data processing circuitry may employ refocusable data of immediatelypreceding and immediately subsequent frames. Indeed, in someembodiments, the immediately preceding frames and/or immediatelysubsequent frames are contiguous. Moreover, the data processingcircuitry may employ certain data acquisition parameters (for example,aperture size) that accelerate or facilitate generation and/or outputvideo to the user interface for Focus Planning consideration by thedevice and/or user/operator. For example, the adjusting of certainacquisition parameters (for example, aperture size or optical focusdepth) may facilitate acquisition of refocusable light field video datathat allows for the generation and/or output video implementing theFocus Planning consideration by the device and/or user/operator (forexample, that allows for implementing the selected, desired orpredetermined virtual focus characteristic or parameter (for example,the selected, desired or predetermined virtual focus depth)).

The selection of focus may be controlled using a variety of differentanalysis methods, all of which are intended to fall within the scope ofthe present inventions. Indeed, all techniques and circuitry forcontrolling the focus selection, whether now known or later developed,are intended to fall within the scope of the present inventions. Severalexamples are presented herein.

Focus Tracking Mode

As noted above, when in the Focus Planning Mode, systems 10 and/or 100or the user/operator may identify one or more subjects/objects(hereinafter collectively “subject” or “subjects”) of interest andtrack, define, maintain and/or set the focus of the refocusable videodata to the one or more subjects. In one embodiment, the Focus PlanningMode allows systems 10 and/or 100 or the user/operator to determinewhether a subject/object is transitory (and, as such, perhaps not ofinterest). A subject to which focus is tracked, defined, maintainedand/or set to may be selected based on, for example, one or more of thefollowing criteria:

-   -   The subject is not part of the background (for example, by using        automatic foreground-background segmentation based on pixel        clustering). In this case, the subject may be more likely to be        selected.    -   The length of time the subject is in the field of view. If the        duration of time that the subject is in the field of view is        very short, the subject may be less likely to be selected.    -   Proximity of the subject to the center of the field of view. The        closer the subject is to the center, the more likely it may be        to be selected.    -   The size of the subject. The larger the subject is, the more        likely it may be to be selected.    -   Whether the subject is or includes a face. Face detection        methods (for example, a suitably configured Haar cascade        classifier) may be used to identify whether the subject region        is a face. If so, it may be more likely to be selected.    -   The identity of the face. If the subject region is a face, it        may be more likely to be selected if a face recognition method        determines that it is someone from a set of known identities.        Any facial recognition method now known or later developed may        be used, including principal component analysis (PCA) or elastic        bunch graph matching (EBGM).    -   Whether the subject is talking or moving (for example, running,        climbing and/or fighting). If the subject is determined to be a        face, it may be more likely to be selected if the person is        talking (for example, by using a detection approach based on        latent semantic indexing) or moving (for example, running,        climbing and/or fighting).

In one exemplary embodiment, the subject tracking operations mayimplement focus transitions that minimize the number of transitions. Inthis way, the output video may appear to be smoother relative to a largenumber of focus transitions. Notably, the system may employ or implementany of the embodiments of focus transition between key frames (asdiscussed herein) in connection with the subject tracking operations.For example, the subject tracking operations may implement a slow focuspull, jump focus pull, peek focus pull manner, form or type of focustransition (as described herein in connection with key frames) duringfocus transitions.

Dialogue Mode

In another exemplary embodiment of the present inventions, when in thedialogue mode, the systems 10 and/or 100 may track, define, maintainand/or set the virtual focus on the subject that is speaking (forexample, by using a talking detection approach based on latent semanticindexing) or moving (for example, running, climbing and/or fighting).Notably, the focus planning may identify between a primary speaker(s)and secondary speakers or others in a group in a scene. In oneembodiment, for example, a primary speaker is a speaker that spends thelargest fraction of the time encompassed by the focus planning timewindow.

Party Mode

In yet another exemplary embodiment of the present invention, in partymode, systems 10 and/or 100 adjusts, defines, maintains and/or sets thevirtual aperture and/or focus to keep all subjects of interest in focus.As more subjects of interest enter the scene, the aperture may bereduced. Subjects of interest may be determined, for example, usingcriteria as described in the Focus Tracking Mode section or any othercriteria now known or later developed.

Live “Is Shot Refocusable?”

As noted above, systems 10 and/or 100 may use the methods or techniquesdescribed in the Live “Is Shot Refocusable” section for indicating tothe operator or user which portions of the field of view are amenable torefocusing after recording.

On-System Editing and Saving

Once the refocusable video data has been captured, acquired and/orobtained, it may be stored in on-system or external memory. A list ofpossible storage formats is set forth above. While video output from theintegrated embodiments of the present inventions may be stored in anyone or more of a plurality of formats (including those listed herein),refocusable video acquisition unit 12 may operate or output a standardvideo data stream. The data stream may be generated fully automaticallyby allowing refocusable video acquisition unit 12 to select, designateand/or choose the subject or focus depth, or alternatively, refocusablevideo acquisition unit 12 may temporarily store refocusable video data(for example, the light field video data) and allow immediate focusediting of the refocusable video data stream.

In the exemplary embodiments described herein, operations, computations,generations, and/or functions of refocusable video acquisition system10, refocusable video editing system 100, and/or components or elementsthereof, may be implemented with processing circuitry disposed (in partor in whole) in/on systems 10 and/or 100 or in/on an external processingsystem. The integrated circuitry may include one or moremicroprocessors, Application-Specific Integrated Circuits (ASICs),digital signal processors (DSPs), and/or programmable gate arrays (forexample, field-programmable gate arrays (FPGAs)). Indeed, the processingcircuitry may be any type or form of circuitry whether now known orlater developed. For example, the processing circuitry may include asingle component or a multiplicity of components (microprocessors, ASICsand DSPs), either active and/or passive, which are coupled together toimplement, provide and/or perform a desiredoperation/function/application.

Further, as mentioned above, in operation, the processing circuitryand/or the circuitry of the systems 10 and/or 100 may perform or executeone or more applications, routines, programs and/or data structures thatimplement particular methods, tasks or operations described andillustrated herein. The functionality of the applications, routines orprograms may be combined or distributed. Further, the applications,routines or programs may be implementing by the processing circuitryusing any programming language whether now known or later developed,including, for example, assembly, FORTRAN, C, C++, and BASIC, whethercompiled or uncompiled code; all of which are intended to fall withinthe scope of the present invention.

Software Autofocus

In yet another exemplary embodiment, refocusable video acquisitionsystem 10 or unit 12 may employ software autofocus which may implementfocus planned recording (see the “Focus Planned Recording” section) orany other method or technique for identifying, determining, definingand/or selecting a virtual focus parameter while refocusable videoacquisition unit 12 captures, acquires and/or obtains refocusable videodata. Under these circumstances, the input refocusable video data streammay be processed to generate a single stream of standard video. Suchstandard video may be stored in on-system or external memory. (See, forexample, FIG. 22). Notably, FIG. 22 relates to an exemplary embodimentof refocusable video acquisition system 10 or unit 12 employing softwareautofocus which reads in a refocusable video data stream and outputsautomatically focused frames.

Buffer and Edit

As noted above, refocusable video acquisition system 10 or unit 12 maybuffer the input refocusable video stream, and thereafter output videowhich is representative thereof to the user interface. In thisembodiment, the user or operator may control the focus selection,aperture size and/or shape and/or exposure with an on-camera edit userinterface. In this way, the user or operator may control the acquisitionparameters (for example, control the focus selection) of the refocusablevideo data. In this embodiment, refocusable video acquisition system 10or unit 12 captures or acquires refocusable video data and temporarilystores such data (for example, stores the full light field video). Thevideo may be saved to a non-volatile storage such as Flash memory orhard drive, or it may be saved in volatile memory, such as DRAM or SRAM.In this embodiment, the system 10 may attempt to initially select thefocus of the video acquisition automatically or it may be left to theuser.

The system 10 and/or 100 may next present the user or operator with aninterface to edit the virtual focus parameter of the refocusable videodata by playing the video back to the user or operator and selecting orallowing selection of the subject of focus. This selection may, forexample, be menu driven or may use a touch screen where the use simplytaps on the subject of focus during the stream. Once the focus has beenselected for the refocusable video data stream, a final stream ofstandard 2D video may be rendered or generated and thereafter providedto storage (for example, non-volatile on-system or external storage),and the original stream of refocusable video may be discarded. Thisembodiment is show in FIG. 23.

There are many inventions described and illustrated herein. Whilecertain embodiments, features, attributes and advantages of theinventions have been described and illustrated, it should be understoodthat many others, as well as different and/or similar embodiments,features, attributes and advantages of the present inventions, areapparent from the description and illustrations. As such, theembodiments, features, attributes and advantages of the inventionsdescribed and illustrated herein are not exhaustive and it should beunderstood that such other, similar, as well as different, embodiments,features, attributes and advantages of the present inventions are withinthe scope of the present inventions.

Moreover, the present inventions are neither limited to any singleaspect nor embodiment thereof, nor to any combinations and/orpermutations of such aspects and/or embodiments. Moreover, each of theaspects of the present inventions, and/or embodiments thereof, may beemployed alone or in combination with one or more of the other aspectsof the present inventions and/or embodiments thereof. For example,systems and/or techniques according to the present inventions may employone or more of the embodiments of the (1) setting the focus based on keyframes in post-production, and/or (2) setting the focus based ontracking subjects, and/or (3) an indicator used during a live shoot thatallows the operator to understand the sharply refocusable range in thescene. Indeed, such embodiments may be implemented on a stand-alonerefocusable video acquisition system and stand-alone refocusable videoediting system and/or an integrated refocusable video acquisition andediting system. For the sake of brevity, many of those permutations andcombinations are not discussed separately herein.

In addition, the exemplary embodiments set forth herein are at timesdescribed in the context where refocusable light field video data.However, it is intended that these exemplary embodiments, although setforth in the context of light field video data, are applicable to andmay utilize any refocusable video data, whether now known or laterdeveloped, where that video data enables computation of images focusedat different depths for various frame of the video. As such, the aboveembodiments of the present inventions are merely exemplary embodiments.They are not intended to be exhaustive or to limit the inventions to theprecise forms, techniques, materials and/or configurations disclosed.

Many modifications, variations combination and/or permutations arepossible in light of the above teaching. It is to be understood thatother embodiments may be utilized and operational changes may be madewithout departing from the scope of the present inventions. As such, theforegoing description of the exemplary embodiments of the inventions hasbeen presented for the purposes of illustration and description. It isintended that the scope of the inventions not be limited solely to thedescription above.

Notably, the storage, memory and/or memory buffer(s) described and/orillustrated herein may be integrated (for example, integrated in theprocessing circuitry) or discrete memory of any kind or type, includingSRAM, DRAM, latches, and/or registers. All memory types, architecturesand forms, and permutations and/or combinations thereof, whether nowknown or later developed, are intended to fall within the scope of thepresent inventions.

Further, as mentioned above, in operation, the processing circuitry mayperform or execute one or more applications, routines, programs and/ordata structures that implement particular methods, tasks or operationsdescribed and illustrated herein (for example, acquiring and/or editingthe refocusable video data and/or generating or rendering output videodata corresponding to refocusable video data using one, some or all ofthe aforementioned editing and/or rendering techniques). The operationsof the applications, routines or programs may be combined ordistributed. Further, the processing circuitry may implement one ormore, or all of such editing and/or rendering techniques in anycombination and all permutations; such techniques may be employed aloneor in combination with one or more of the other techniques of acquiringand/or editing the refocusable video data and/or generating or renderingoutput video data corresponding to refocusable video data. Thetechniques, methods and/or applications may be implemented by theprocessing circuitry using any programming language whether now known orlater developed, including, for example, assembly, FORTRAN, C, C++, andBASIC, whether compiled or uncompiled code; all of which are intended tofall within the scope of the present invention.

It should be noted that the term “circuit” may mean, among other things,a single component (for example, electrical/electronic) or amultiplicity of components (whether in integrated circuit form orotherwise), which are active and/or passive, and which are coupledtogether to provide or perform a desired function. The term “circuitry”may mean, among other things, a circuit (whether integrated orotherwise), a group of such circuits, one or more processors, one ormore state machines, one or more processors implementing software, oneor more gate arrays, programmable gate arrays and/or field programmablegate arrays, or a combination of one or more circuits (whetherintegrated or otherwise), one or more state machines, one or moreprocessors, one or more processors implementing software, one or moregate arrays, programmable gate arrays and/or field programmable gatearrays. The term “data” may mean, among other things, a current orvoltage signal(s) (plural or singular) whether in an analog or a digitalform, which may be a single bit (or the like) or multiple bits (or thelike).

It should be further noted that the various circuits and circuitrydisclosed herein may be described using computer aided design tools andexpressed (or represented), as data and/or instructions embodied invarious computer-readable media, in terms of their behavioral, registertransfer, logic component, transistor, layout geometries, and/or othercharacteristics. Formats of files and other objects in which suchcircuit expressions may be implemented include, but are not limited to,formats supporting behavioral languages such as C, Verilog, and HLDL,formats supporting register level description languages like RTL, andformats supporting geometry description languages such as GDSII, GDSIII,GDSIV, CIF, MEBES and any other suitable formats and languages.Computer-readable media in which such formatted data and/or instructionsmay be embodied include, but are not limited to, non-volatile storagemedia in various forms (e.g., optical, magnetic or semiconductor storagemedia) and carrier waves that may be used to transfer such formatteddata and/or instructions through wireless, optical, or wired signalingmedia or any combination thereof. Examples of transfers of suchformatted data and/or instructions by carrier waves include, but are notlimited to, transfers (uploads, downloads, e-mail, etc.) over theInternet and/or other computer networks via one or more data transferprotocols (e.g., HTTP, FTP, SMTP, etc.).

Indeed, when received within a computer system via one or morecomputer-readable media, such data and/or instruction-based expressionsof the above described circuits may be processed by a processing entity(e.g., one or more processors) within the computer system in conjunctionwith execution of one or more other computer programs including, withoutlimitation, net-list generation programs, place and route programs andthe like, to generate a representation or image of a physicalmanifestation of such circuits. Such representation or image maythereafter be used in device fabrication, for example, by enablinggeneration of one or more masks that are used to form various componentsof the circuits in a device fabrication process.

1. A method of generating and outputting video data corresponding to aplurality of video data frames, the method comprising: receivingrefocusable light field video data, wherein the refocusable light fieldvideo data includes a plurality of temporally contiguous refocusablelight field video frames, wherein each refocusable light field videoframe includes an optical focus; selecting a first key frame, whereinthe first key frame corresponds to one of the plurality of refocusablelight field video frames; determining a virtual focus parameter for thefirst key frame; selecting a second key frame, wherein the second keyframe corresponds to one of the plurality of refocusable light fieldvideo frames which is temporally spaced apart from the first key framesuch that a plurality of refocusable light field video frames aretemporally disposed between the first and the second key frames;determining a virtual focus parameter for the second key frame;generating first video data corresponding to the plurality ofrefocusable light field video frames which are temporally disposedbetween the first and the second key frames using (i) the virtual focusparameter for the first key frame, (ii) the virtual focus parameter forthe second key frame and (iii) the refocusable light field video datacorresponding to the plurality of refocusable light field video frameswhich are temporally disposed between the first and the second keyframes, wherein the first video data includes a plurality of videoframes, each video frame including a virtual focus depth which is basedon the virtual focus parameter for the first key frame and/or virtualfocus parameter for the second key frame; and outputting the first videodata.
 2. The method of claim 1 wherein the virtual focus parameter ofthe first key frame and the virtual focus parameter of the second keyframe each include information which is representative of a virtualfocus depth.
 3. The method of claim 1 wherein the virtual focus depthcorresponding to the virtual focus parameter of the first key frame isdifferent from the optical focus depth of the refocusable light fieldvideo frame corresponding to the first key frame.
 4. The method of claim1 wherein the virtual focus parameter of the first key frame includesinformation which is representative of focus tracking.
 5. The method ofclaim 1 wherein the virtual focus parameter of the first key frame andthe virtual focus parameter of the second key frame each includeinformation which is representative of a location of a virtual focalplane, aperture and/or focus effect.
 6. The method of claim 1 whereinthe virtual focus parameter of the first key frame includes informationwhich provides a tilted virtual focal plane.
 7. The method of claim 1further including: generating final output video data by processing thefirst video data to provide compositing and/or transitions and/orfiltering and/or color adjustments of a plurality of the video framescorresponding to the first video data.